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Abstract 


Although  several  techniques  have  been  developed  to 
enhance  R&D  project  selection  decision-making  in  research 
laboratories,  they  have  not  been  widely  accepted  by 
managers.  This  research  investigated  three  aspects 
relating  to  the  implementation  of  R&D  project  selection 
techniques  at  the  Air  Force  Wright  Aeronautical  Labora¬ 
tories  (AFWAL) .  It  examined  1)  the  methods  used  by 
managers  to  make  R&D  project  selection  decisions;  2)  the 
attitudes  and  perceptions  of  managers  toward  the  utility 
of  R&D  project  selection  techniques;  and  3)  the  specific 
factors  affecting  the  adoption  of  these  techniques  by  R&D 
managers . 

Findings  showed  that  formal  R&D  project  selection 
techniques  play  a  minor  role  in  the  project  selection 
process,  and  that  managers  are,  in  general,  unfamiliar 
with  a  majority  of  these  techniques.  It  was  found  that 
project  selection  decisions  are  made  by  a  "corporate 
board"  consisting  of  top-level  managers.  The  respondents 
indicated  that  they  were  generally  satisfied  with  this 
process  despite  feeling  a  moderate  amount  of  pressure  to 
quantitatively  justify  their  resource  expenditures. 
Findings  also  showed  that  the  R&D  project  selection 
process  at  AFWAL  is  heavily  influenced  by  "requirements 
pull"  as  opposed  to  "technology  push." 


viii 


Factor  analysis  was  used  to  identify  an  initial  set 
of  variables  which  affect  management's  willingness  to 
adopt  formal  project  selection  techniques.  Nine 
variables  were  found  to  affect  management's  "willingness 
to  adopt."  These  variables  were  factored  into  two 
groups  which  relate  to  1)  the  effect  of  project  selection 
techniques  on  organizational  decision-making;  and  2)  the 
characteristics  of  the  project  selection  techniques 
themselves.  The  results  showed  that  the  respondents 
possessed  negative  perceptions  toward  the  utility  of 
these  techniques  as  a  management  tool. 

Previous  research  efforts  have  shown  that  negative- 
to-positive  attitude  shifts  occurred  after  respondents 
were  educated  and  trained  in  the  use  of  these  tech¬ 
niques.  Consequently,  it  was  recommended  that  the 
respondents  be  provided  with  training  to  increase  their 
understanding  of  these  techniques.  Once  accomplished,  a 
follow-up  survey  should  be  administered  to  assess  the 
stability  of  the  initial  set  of  adoption-affecting 
variables,  and  the  results  used  as  a  guide  to  select  or 
develop  project  selection  techniques  which  would  be 
accepted  and  implemented  as  a  management  tool. 
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FACTORS  AFFECTING  THE  ADOPTION  OF  R&D  PROJECT  SELECTION 
TECHNIQUES  AT  THE  AIR  FORCE  WRIGHT  AERONAUTICAL  LABORATORIES 

I.  Introduction 

Background 

Within  the  Air  Force,  the  continued  advancement  and 
expansion  of  technological  capabilities  is  the  respon¬ 
sibility  of  a  network  of  research  and  development  (R&D) 
laboratories  operating  under  the  direction  of  the  Air  Force 
Systems  Command.  This  network  of  laboratories  is  the  focal 
point  for  the  planning  and  execution  of  basic  research, 
exploratory  and  advanced  development,  selected  engineering 
development,  and  assigned  manufacturing  technology  programs 
(32:2).  The  R&D  manager  in  this  environment  is  responsible 
for  evaluating,  selecting,  and  funding  research  projects 
which  exhibit  a  high  potential  for  achieving  technical 
success.  This  process  of  R&D  project  selection  and  resource 
allocation  has  been  called  "the  key  to  research  success" 
(33:169) . 

In  an  effort  to  improve  the  effectiveness  of  this 
decision-making  process,  several  formal  project  selection 
techniques  have  been  developed.  Ranging  from  scoring  models 
to  linear  programming,  these  techniques  are,  for  the  most 
part,  derived  from  applications  of  management  science  and 
operations  research  (MS/OR)  (15:16). 
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Ideally,  these  project  selection  techniques  offer  the 
R&D  manager  a  systematic  method  for  assessing  the  potential 
value  of  research  projects,  thereby  improving  managerial 
decision  making.  However,  surveys  of  both  private  and 
public  sector  research  institutes  indicate  that  R&D  managers 
are  reluctant  to  adopt  these  techniques  (23:962;  31:40). 
Methods  to  increase  the  adoption  of  MS/OR  as  a  management 
tool  have  been  the  subject  of  focused  research  since  the 
early  1970s  (39:xiii). 

Management's  reluctance  toward  the  use  of  MS/OR  is  not 

unique  to  the  R&D  environment.  The  implementation  of 

quantitative  techniques  has  been  found  to  be  quite  low  for 

many  potential  applications  (40: 3) .  Surprisingly,  this 

reluctance  on  the  part  of  managers  has  persisted,  despite 

increased  exposure  to  MS/OR  in  academia,  and  the  development 

of  user-friendly  software  for  project  management  (23:963). 

Randall  Schultz  and  Dennis  Slevin  have  expressed  concern 

about  this  trend,  noting  that: 

The  ultimate  question  that  can  be  asked  of  management 
science  is  whether  or  not  it  has  had  an  impact  on 
decision  making  in  organizations.  The  question 
involves  management  science  implementation,  the 
bridging  of  the  gap  between  theory  and  practice,  and 
the  improvement  of  organizational  effectiveness  through 
organizational  change.  (40:1) 

Justification 

In  "Our  Tech  Base  Needs  Attention, "  General  Robert  Marsh 
underscores  the  importance  of  maintaining  U.S.  technical 


2 


superiority  to  counter  Soviet  military  challenges.  According 
to  General  Marsh, 

if  we  seek  to  maintain  our  security  without  having  to 
pay  the  prohibitive  price  of  matching  the  Soviet  Union 
in  sheer  numbers,  then  we  must  resolve  to  adequately 
support  the  research  and  development  necessary  to  stay 
ahead  technologically.  (28:90) 

General  Marsh  notes  that  an  unfavorable  trend  is 
developing  in  the  balance  between  U.S.  and  Soviet  techno¬ 
logical  development  which  is  "a  direct  result  of  diminished 
U.S.  support  for  military  R&D"  (28:91).  Thus,  in  the 
future,  the  military  R&D  manager  will  be  faced  with  the 
challenge  of  reversing  this  trend  with  a  constrained  research 
budget.  As  such,  it  will  be  of  paramount  importance  that 
laboratory  resources  are  used  as  effectively  and  efficiently 
as  possible.  Unfortunately,  the  implementation  of  techniques 
designed  to  enhance  R&D  project  selection  decision  making  has 
been  found  to  be  a  difficult  and  complex  process. 

In  Management  -  A  Quantitative  Perspective.  N.  Paul 
Loomba  examines  the  nature  of  the  implementation  problem: 

The  implementation  of  quantitative  models  is  an 
extremely  complex  area.  It  involves  multifaceted 
relationships  and  interactions  among  the  manager, 
management  scientist,  top  administration, 
production  personnel,  and  other  organizational 
resources  (e.g.  computer  systems) .  (25:457) 

Thus,  the  goal  of  the  implementation  researcher  is  to 

gain  an  understanding  of  the  attitudes,  perceptions,  and 

requirements  of  managers  concerning  MS/OR  techniques.  With 

this  information,  MS/OR  professionals  can  develop  techniques 
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that  are  responsive  to  the  needs  of  managers,  thus  increasing 

their  acceptance  by  organizations. 

By  1968  it  was  recognized  that  there  was  a  need  for  a 

mutual  understanding  between  those  who  develop  MS/OR 

techniques  -  the  management  scientists  -  and  the  managers  who 

may  utilize  these  techniques.  Thad  Green  notes  that  progress 

in  this  area  has  been  slow: 

More  than  a  decade  has  passed  and  there  is  little 
progress  toward  a  mutual  understanding.  This  is 
due  to  the  very  nature  of  the  situation.  Managers 
who  do  not  see  the  value  in  using  decision  science 
models  are  reluctant  to  use  them.  Yet,  until 
managers  use  the  models,  they  likely  will  not 
perceive  them  as  having  much  value.  Thus  we  are 
in  a  vicious  cycle.  (14:6) 

The  need  for  continued  implementation  research  is 
readily  apparent.  In  a  review  of  implementation  literature, 
Schultz  and  Slevin  concluded  that  1)  there  has  been  an 
insufficient  amount  of  research  in  the  area;  and  2)  what  has 
been  done  is  theoretical  in  nature,  as  opposed  to  empirical 
(38:153) . 

Although  theoretical  discussions  are  a  necessary 
and  integral  part  of  the  development  of  any  field 
of  research,  at  some  point  in  time,  they  must  be 
examined  for  their  conformity  with  empirical 
evidence.  (38:153) 


Application?  in  the  R&P  Environment 

While  it  may  appear  that  R&D  managers  would  be  receptive 
to  the  "scientific  method"  upon  which  formal,  sophisticated 
project  selection  techniques  are  based,  the  evidence  does  not 
support  this  assumption.  A  1983  industry  survey  of  40 
laboratory  managers  from  29  Fortune  500  companies  concluded 
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that  there  was  ’’heavy  use  of  financial  methods  of  project 

selection,  but  virtually  no  use  of  math  programming  for  R&D 

resource  allocation”  (23:962). 

In  addition,  a  1985  survey  of  managers  at  the  Air  Force 

Wright  Aeronautical  Laboratories  (AFWAL)  found  that  only  30 

percent  of  the  respondents  reported  using  a  formal  project 

selection  technique  (31:40).  Dr.  William  Souder  notes  that 

this  lack  of  usage  may  be  a  consequence  of  the  lack  of 
attention  which  model  builders  have  traditionally  given 
to  the  prevailing  adoption  attitudes  of  R&D  managers. 
(45:75) 

A  comparison  of  the  R&D  environments  in  both  the  public 
and  private  sectors  sheds  additional  light  on  the  implemen¬ 
tation  problem. 

Whereas  private  sector  R&D  is  primarily  concerned  with 
profit  maximization,  the  goal  of  military  R&D  is  to  enhance 
our  ability  to  meet  mission  requirements.  The  complexities 
of  the  military  mission  and  the  focus  of  military  research  on 
operational  enhancement  places  unique  constraints  on  the 
military  R&D  manager.  Unfortunately,  many  of  the  project 
selection  techniques  which  are  designed  to  assist  managers  in 
the  selection  of  research  projects  were  developed  for  the 
private  sector,  and  with  their  objective  of  profit  maximiza¬ 
tion  in  mind.  This  makes  their  utility  questionable  in  the 
context  of  public  sector  R&D  (36:126). 

In  addition,  there  are  many  organizational  differences 
between  the  public  and  private  sectors.  In  ”The  Future  of 
Operations  Research  in  the  Government,”  Thomas  Saaty  offers 
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this  explanation  for  the  apparent  resistance  within  the 

government  toward  the  use  of  MS/OR  techniques: 

Departments  of  the  government  are  rarely  in 
competition  for  survival,  though  their  leaders  may 
come  under  political  fire.  For  their  structure, 
they  rely  on  a  long  tradition  of  operation.  It  is 
natural  that  they  tend  to  resist  change  to 
maintain  their  identity.  Sometimes  rivalry  and 
power  plays  lead  to  rigidity.  Undoing  these  rigid 
forms  is  a  slow  process.  Industry  operates  in  a 
competitive  environment.  To  survive,  it  must  be 
flexible.  (37:6) 

Despite  these  difficulties,  there  are  some  examples  of 

successful  implementations  of  quantitative  project  selection 

techniques  in  government  research  agencies.  The  Division  of 

Power  Systems  of  the  Department  of  Energy  researched  formal 

project  selection  techniques  in  1978  and  subsequently  adopted 

a  scoring  model  to  assist  in  the  selection  of  applied  R&D 

projects  (3:1).  In  addition,  new  advances  in  formal  project 

selection  techniques  appear  to  offer  the  military  R&D  manager 

a  rational,  systematic  method  for  assessing  research  projects 

and  allocating  resources.  Souder  notes  the  following: 

The  naive  modelling  application  philosophies  of 
the  1950s  and  1960s  have  grudgingly  given  way  to 
a  more  mature,  modern  philosophy.  Both  managers 
and  model  builders  should  be  aware  of  this 
philosophy  and  its  potential  for  importing 
decisions  about  project  selection.  (46:36) 


general  lasiae 

The  focus  of  this  research  is  the  R&D  project  selection 
process  at  the  Air  Force  Wright  Aeronautical  Laboratories 
(AFWAL)  located  at  Wright-Patterson  Air  Force  Base.  AFWAL 
consists  of  the  Aero  Propulsion  Laboratory,  the  Avionics 
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Laboratory,  the  Flight  Dynamics  Laboratory,  and  the  Materials 
Laboratory.  Together,  these  four  laboratories  have  an  annual 
operating  budget  in  excess  of  $850  million,  and  they  employ 
over  1700  scientists.  AFWAL  is  one  of  the  largest  research 
and  development  complexes  in  the  free  world  (32:2). 

This  research  will  investigate  the  R&D  project  selection 
process  at  AFWAL.  It  will  focus  on  1)  the  methods  used  by 
AFWAL  managers  to  select  R&D  projects  and  allocate  laboratory 
resources;  2)  the  attitudes  and  perceptions  of  AFWAL  managers 
toward  the  utility  of  R&D  project  selection  techniques;  and 
3)  the  specific  factors  that  affect  management's  willingness 
to  adopt  these  techniques. 

The  results  of  this  research  will  provide  valuable 
insight  into  the  R&D  project  selection  process  and  organiza¬ 
tional  climate  at  AFWAL.  By  measuring  management's  percep¬ 
tions  toward  the  utility  of  formal  project  selection 
techniques,  and  subsequently  deriving  a  set  of  adoption- 
affecting  variables,  a  basis  is  formed  from  which  to  evaluate 
project  selection  techniques  and  identify  those  which  are 
compatible  with  the  organization  (44:133;  45:75). 

Research  Questions 

The  following  research  questions  will  be  used  to  guide 
this  investigation: 

1.  What  methods  do  managers  use  to  select  R&D  projects 
and  allocate  laboratory  resources  at  the  Air  Force 
Wright  Aeronautical  Laboratories? 
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2.  What  are  the  attitudes  and  perceptions  of  managers 
toward  the  use  of  formal  R&D  project  selection 
techniques? 

3.  What  factors  affect  management's  willingness  to 
adopt  formal  R&D  project  selection  techniques? 

Scope  and  Limitations 

This  research  focused  primarily  on  AFWAL  as  a  single 
organization.  However,  whenever  possible,  significant 
differences  between  the  individual  laboratories  which 
comprise  AFWAL  were  noted. 

While  there  are  several  other  laboratories  operating 
under  the  direction  of  the  Air  Force  Systems  Command,  time 
and  travel  constraints  made  it  impractical  to  examine  these 
facilities. 

Terms  -and_Jtef  inifrlong 

For  the  purpose  of  this  research,  the  following  terms 
and  operational  definitions  will  be  used: 

R&D  project  selection  techniques.  "A  formal,  often 
quantitative  model  or  algorithm  employed  to  assign  values  to 
individual  R&D  projects  or  groups  of  projects"  (36:124). 

For  this  research,  the  terms  "project  selection  technique," 
"formal  technique,"  "project  selection  model,"  "quantitative 
technique,"  and  "formal  decision  method"  are  synonymous. 

Implementation .  The  involvement  of  a  formal  project 
selection  technique  to  aid  the  process  of  evaluating 
competing  R&D  projects  and  allocating  resources  (40:6). 
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Management  science/operations  research  CMS/OR) .  A 
branch  of  the  field  of  management  which  employs  a  rational, 
logical,  systematic,  and  scientific  approach  to  analyze  the 
process  of  management  and  management  problems  (25:18).  For 
the  purpose  of  this  research,  the  terms  "management  science," 
"operations  research,"  "decision  sciences,"  "system  sci¬ 
ences,"  and  "systems  analysis"  will  be  used  interchangeably. 

Chapter  Summary 

This  chapter  provided  an  introduction  to,  and  motivation 
for  this  research  effort.  It  began  with  a  discussion  of  the 
implementation  of  MS/OR  and  R&D  project  selection  techniques 
in  organizations.  Next,  the  need  for  empirical  research  in 
this  area  was  established.  This  was  followed  by  an  overview 
of  the  R&D  environment  in  both  the  public  and  private  sector. 
Finally,  the  research  questions  that  guided  this  effort  were 
presented,  along  with  a  series  of  key  definitions.  The 
following  chapter  provides  a  review  of  applicable  literature. 
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This  literature  review  examines  three  major  topic  areas. 
In  the  first  section.  Implementation  Research,  a  working 
definition  of  implementation  is  provided,  as  well  as  a  review 
of  the  origins  of  MS/OR  as  a  management  tool.  The  remainder 
of  the  section  is  devoted  to  a  review  of  several  prominent 
implementation  research  efforts.  The  research  efforts 
included  in  this  review  were  selected  for  their  contribution 
to  the  identification  of  factors  affecting  the  implementation 
of  MS/OR  based  techniques  in  a  variety  of  organizations. 

The  second  section,  The  Implementation  of  R&D  Project 
Selection  Techniques,  is  more  narrowly  focused,  and  specif¬ 
ically  examines  the  implementation  of  R&D  project  selection 
techniques.  This  section  includes  a  discussion  of  the 
barriers  that  often  preclude  the  use  of  formal  R&D  project 
selection  techniques.  It  also  includes  a  comparison  of  R&D 
management  in  the  public  and  private  sectors. 

The  third  section,  R&D  Project  Selection  Techniques, 
examines  four  categories  of  R&D  project  selection  techni¬ 
ques.  This  section  provides  a  review  of  each  category's 
capabilities,  strengths,  and  weaknesses. 

Implementation  Research 

Implementation  Defined.  A  wide  variety  of  definitions 
are  used  throughout  the  literature  to  describe  the  implemen¬ 
tation  of  MS/OR  in  organizations.  This  research  is  best 
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served  by  the  "behavioral  perspective  of  implementation" 
proposed  by  Schultz  and  Slevin  (40:3).  Schultz  and  Slevin 
note  that  there  are  three  major  participants  in  the  implemen¬ 
tation  process:  MS/OR  professionals,  middle  management,  and 
top  management.  The  definition  developed  by  Schultz  and 
Slevin  considers  the  perspective  of  each  of  these  partic¬ 
ipants  . 

The  perspective  of  the  MS/OR  professional  is  that  if  a 
technique  works,  and  is  technically  valid,  it  is  successful. 
On  the  other  hand,  middle  managers  take  the  point  of  view 
that  a  technique  is  successful  only  if  it  is  used  and  main¬ 
tained  by  the  organization.  Finally,  top  management  requires 
not  only  that  a  MS/OR  technique  is  used  and  maintained,  but 
that  it  influences  decision  making  in  a  way  that  benefits  the 
organization  (40:3).  To  fully  define  implementation  in  the 
context  of  the  organization,  all  three  perspectives  must  be 
integrated.  According  to  Schultz  and  Slevin, 

the  behavioral  perspective  of  implementation  includes 
the  development  (technical  validity)  and  the  use 
(organizational  validity)  of  a  model  or  technique  that 
results  in  a  positive  change  in  organizational  effec¬ 
tiveness  (positive  top  management  evaluation).  (40:4) 

A  Historical  Perspective.  MS/OR  emerged  during  World 

War  II,  and  rapidly  gained  in  popularity.  From  this  time, 

until  the  mid-1960s,  MS/OR  was  seen  as  a  new  and  effective 

management  tool.  Its  use  was  widely  accepted  and  encouraged 

by  many  top  managers  in  both  industry  and  government. 

However,  in  the  years  following,  some  MS/OR  applications  did 

not  live  up  to  expectations.  Subsequently,  top  management 
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acceptance  of  MS/OR  began  to  decline.  This  prompted  several 
research  efforts  to  determine  its  causes  (40:5). 

Current  Research  Efforts.  In  1973,  a  conference 
entitled  "The  Implementation  of  OR/M S  Models:  Theory, 
Research,  and  Application"  was  held  in  Pittsburgh,  Pennsyl¬ 
vania.  Its  purpose  was  to  review  previous  implementation 
research  and  "identify  avenues  that  will  lead  to  a  better 
utilization  of  MS/OR  as  a  true  problem-solving  activity" 
(39:xiii).  This  conference  served  to  consolidate  and  focus 
implementation  research  efforts. 

Since  1973,  there  has  been  a  significant  increase  in 
implementation  research  efforts.  Several  of  the  most 
prominent  research  efforts  of  the  past  15  years  have  been 
compiled  into  three  major  works  entitled  Implementing 
Operations  Research/Management  Science,  published  in  1975, 

The  Ipplgmanta-tigp  si.  Management  Ssisnss,  published  in  1979, 
and  Applications _a£ JlanagSlBSDfc  SSisnSS ,  published  in  1984. 
Though  these  compiled  works  represent  three  distinct 
"generations"  of  implementation  research,  they  each  emphasize 
the  behavioral  aspects  of  the  implementation  process  (41 :x) . 

The  principal  reason  for  this  reliance  on  behavioral 
methodologies  is  the  fundamental  nature  of  the  imple¬ 
mentation  problem.  It  is,  in  any  interpretation,  a 
problem  involving  human  participants,  social  interac¬ 
tion,  organizational  structure,  and  management  change  - 
in  short,  a  complex  behavioral  process.  (40:8) 

The  First  Generation.  The  first  generation  of 
implementation  research  can  be  characterized  as  an  effort  to 
model  the  implementation  process  in  organizations  (42:63). 
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These  research  efforts  often  resulted  in  the  identification 

of  variables  which  were  found  to  influence  the  implementation 

of  MS/OR  techniques  in  organizations. 

The  research  conducted  by  Souder,  and  presented  in  "An 

Organizational  Intervention  Approach  to  the  Design  and 

Implementation  of  R&D  Project  Selection  Models,"  provided  the 

framework  for  this  effort  (44;  45).  Souder's  research 

methodology  is  described  below. 

The  methodology  consists  of  procedures  for  analyzing  the 
organizational  climate  relative  to  project  selection 
model  usage,  developing  an  acceptable  model  form 
relative  to  the  organization,  and  inducing  the  adoption 
of  this  model  form.  (45; 75) 

To  facilitate  this  research,  Souder  developed  a  Likert- 
based  survey  instrument  designed  to  test  several  variables 
which  were  thought  to  influence  a  manager's  willingness  to 
adopt  R&D  project  selection  techniques.  Using  correlation 
techniques,  Souder  found  that  the  adoption-affecting 
variables  could  be  grouped  into  three  factors;  these  factors 
are  "model  characteristics,"  "organizational  factors,"  and 
"personal  decision  variables"  (44:139).  Figure  1  depicts 
the  variables,  and  how  they  were  grouped  into  the  factors 
described  above.  The  adoption-affecting  variable  set  was 
then  used  as  an  input  to  develop  a  "linear  discriminant 
function  to  evaluate  and  rate  the  acceptability  or  adoption 
potential"  of  a  variety  of  project  selection  techniques 
(45:79) . 
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Organizational  factor* 


To  ensure  that  the  variables  used  to  evaluate  and  rate 
potential  techniques  accurately  represented  the  views  of  the 
respondents,  Souder  employed  a  longitudinal  research 
approach.  First,  a  survey  was  administered  to  each  respon¬ 
dent  to  obtain  an  adoption-affecting  variables  set.  Next,  a 
training  session  was  provided  to  the  respondents  which 
covered  the  potential  applications  of  R&D  project  selection 
techniques.  Following  the  training  session,  the  survey  was 
re-administered,  and  the  resulting  set  of  adoption-affecting 
variables  was  compared  to  the  original  set.  Souder  maintains 
that  "a  stable  set  of  variables  will  be  indicated  when  two 
successively  administered  questionnaires  do  not  differ 
statistically"  (45:79).  Souder's  longitudinal  approach 
served  two  purposes.  Not  only  did  it  derive  a  stable  set  of 
adoption-affecting  variables,  but  it  also  demonstrated  the 
effect  that  information  and  training  had  on  the  perceptions 
and  attitudes  of  the  respondents.  In  applications  of  this 
research,  Souder  observed  a  "negative-to-positive  shift  in 
adoption  attitudes,"  indicating  that  the  lack  of  under¬ 
standing  of  the  capabilities  of  R&D  project  selection 
techniques  has  a  significant  influence  on  their  adoption  by 
management . 

Schultz  and  Slevin  have  also  made  significant  contribu¬ 
tions  to  implementation  research.  One  of  their  more 
noteworthy  efforts  is  described  in  "Implementation  and 
Organizational  Validity:  An  Empirical  Investigation" 

(38:153).  In  this  effort,  the  researchers  sought  to 
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determine  the  key  factors  which  affect  the  successful 
implementation  of  MS/OR  in  organizations.  The  researchers 
found  that  the  respondents'  attitudes  toward  the  implemen¬ 
tation  of  MS/OR  techniques  were  dependent  upon  several 
factors.  These  factor  are  summarized  in  Table  1. 

Table  1. 

Schultz  and  Sievin' s  Factors 
Affecting  Implementation  Success 

1.  Effect  on  job  performance 

2.  Effect  on  interpersonal  relationships 

3.  Extent  of  organizational  change 

4.  Effect  on  organizational  goals 

5.  Degree  of  top  management  support 

6.  Implementor  and  manager  relations 

7.  The  urgency  and  need  for  an  MS/OR  technique 

Adapted  from  (38:153) 

The  research  conducted  by  John  Manley  has  also  contrib¬ 
uted  significantly  to  the  identification  of  adoption- 
affecting  variables.  Manley  focused  on  the  measurement  of 
external  variables  associated  with  MS/OR  implementation. 

In  Manley's  experiment,  the  subjects  were  given  nearly 
identical  descriptions  of  an  MS/OR  model.  However,  one  of 
the  key  variables  shown  in  Table  2  was  changed  in  each  new 
description.  From  these  responses,  Manley  measured  the 
group's  resistance  to,  or  support  of  the  MS/OR  model,  in 
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terms  of  the  five  key  variables  (26:190).  These  variables 
are  shown  in  Table  2. 

Table  2. 

Manley's  External  Variables 
Affecting  MS/OR  Implementation 

1.  The  level  of  chief  executive  support  for  the  MS/OR 
product . 

2.  The  priority  level  assigned  to  the  implementation 
project  by  management. 

3.  The  degree  of  product  relevancy  to  the  client's 
organizational  role. 

4.  The  level  of  required  client  involvement  to 
implement  the  product 

5.  The  complexity  of  the  MS/OR  product. 

Adapted  from  (26:190) 

Several  research  efforts  conducted  at  Northwestern 
University  under  the  direction  of  Michael  Radnor  have  also 
provided  a  significant  contribution  to  the  field  of  implemen¬ 
tation  research.  Table  3  shows  the  variables  which  were 
found  by  Radnor  and  the  researchers  at  Northwestern  Univer¬ 
sity  to  influence  the  implementation  of  MS/OR  in  organ¬ 
izations. 

Gary  Powell's  work  also  enhanced  implementation  research 

considerably.  In  his  efforts,  Powell  maintained  that  the 

failure  of  MS/OR  in  organizations  was  due  to  the  lack  of 

consideration  of  "human  variables"  in  the  model-building 

process  (30:88).  Powell  suggests  that 

the  strong  research  orientation  of  MS/OR  has  led  to 
many  projects  being  conducted  without  concern  for 
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whether  the  results  will  ever  be  implemented  in  an 
organizational  setting.  As  a  result,  we  have  more 
sophisticated  technical  models  without  the  improved 
sophistication  needed  to  implement  them.  It  is  evident 
that  problems  of  OR/MS  implementation  may  be  alleviated 
by  more  attention  being  paid  to  human  variables  in 
research  projects  and  to  the  relationship  with  the 
client.  (30:88) 


Table  3 . 

Radnor’s  Variables  Affecting  MS/OR 
Implementation  (circa  1970) 

1.  Recognition  of  the  need  for  an  MS/OR  technique 

2.  Willingness  of  the  individuals  in  the  receiving 
unit  to  interrupt  ongoing  work 

3.  Technical  mismatch  in  understanding  of  the 
specifications  of  the  item 

4.  Mismatch  in  understanding  of  the  objectives  of 
the  project  or  task 

5.  Pre-existing  relationships  of  trust  between  the 
parties  to  an  implementation  transaction 

6.  Degree  of  managerial  involvement  in  the  stages 
of  a  project 

7.  Self-interest 

8 .  Urgency 

9 .  Perceived  threat 

10.  Level  of  managerial  support 

11.  Point  in  time  at  which  a  management  commitment 
is  made  to  the  project 

Adapted  from  (35:971) 


In  his  research,  Powell  found  that  the  six  variables 
shown  in  Table  4  were  the  keys  to  determining  success  or 
failure  of  MS/OR  implementation. 
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Table  4. 


Powell's  Variables  Affecting  MS/OR  Implementation 

1.  The  environmental  context  of  the  organization 

2.  The  goals  and  norms  of  the  organization  itself 

3.  The  relationship  between  the  organization  and  the 
OR/MS  implementation  group 

4.  The  conduct  and  selection  of  projects 

5.  The  individuals  involved 

6.  The  relationship  between  managers  and  operations 
researchers 

Adapted  from  (30:84) 

The  Second  Generation.  Whereas  the  first  genera¬ 
tion  of  implementation  research  was  characterized  by  efforts 
to  model  the  implementation  process,  the  second  generation 
was  noted  for  its  focus  on  empirical  research  in  an  effort  to 
validate  the  previous  models  of  the  implementation  process 
(42:62) . 

Of  particular  significance  to  this  research  is  the 

institutional  analysis  approach  taken  by  Radnor.  In  "The 

Context  of  OR/MS  Implementation,"  Radnor  found  that  the 

weakness  of  much  of  the  existing  implementation  research  is 

its  reliance  on  "narrowly  defined  OR/MS  practitioner/client 

relationship  models"  (34:17).  According  to  Radnor, 

the  literature  has  generally  stopped  with  expressions 
of  philosophy  or  descriptions  of  effects,  and  thus 
fails  to  provide  tools  or  approaches  which  could  help 
MS/OR  practitioners  deal  with  their  real  life  implemen¬ 
tation  problems.  (34:18) 
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Radnor  recommended  future  research  which  would  consider 


a  broader  contextual  approach.  According  to  Radnor, 
implementation  research  must  consider  "the  totality  of  all 
factors  and  dynamics  and  their  interaction  which  impact  OR/MS 
implementation"  (34:18).  Radnor  notes  that 


this  approach  suggests  a  rich,  descriptive  taxonomy  of 
variables  and  an  analytical  methodology  which  will 
allow  OR/MS  implementation  research  to  focus  on  the 
OR/M S  implementation  context,  thereby  stimulating  and 
assisting  in  the  building  of  richer  and  more  realistic 
models.  (34:18) 

Radnor  contends  that  the  contextual  dimensions  shown  in 
Table  5  have  a  critical  effect  on  MS/OR  implementation. 

Table  5. 

Radnor's  Contextual  Dimensions  Affecting 
MS/OR  Implementation  (circa  1979) 

1.  OR/M s  programs:  projects  and  portfolios 

2.  OR/MS  as  a  change  phenomenon 

3 .  Environment 

4.  Organization 

5.  Operations  (technology) 

6 .  Management 

7 .  Resources 

Adapted  from  (34:18) 

Radnor  provides  a  "checklist"  of  factors  to  be  con¬ 
sidered  under  each  of  the  seven  contextual  dimensions  listed 
above,  and  suggests  the  use  of  a  matrix  format  for  indi¬ 
vidually  addressing  each  dimension.  In  total,  Radnor  lists 
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over  one  hundred  individual  factors  which  can  affect  the 
implementation  of  an  MS/OR  project  (34:22-32). 

The  Third  Generation.  The  third  generation  of 
implementation  research  begins  with  the  works  found  in 
Applications  of  Management  Science  (41:ix).  The  editors  of 
this  work  suggest  that  implementation  research  can  be 
categorized  into  three  groups:  factor  research,  cognitive 
research,  and  process  research  (41:x) .  The  authors  note  that 
factor  research  focuses  on  "the  characteristics  of  the 
organization,  problem,  technology,  and  people  involved" 

(41 :x) .  The  second  group,  cognitive  research,  considers  the 
"differences  in  cognitive  styles  between  system  users  and 
builders"  (41:x) .  Finally,  process  research  focuses  on  the 
characteristics  common  to  successful  implementations  (41 :x) . 

With  respect  to  the  research  performed  in  previous 
"generations"  Randall  Schultz  and  Michael  Ginzberg  note  the 
following: 

The  models  used  to  guide  first  and  second  generation 
implementation  research  are,  in  general,  quite  simple. 
Thus  they  have  only  limited  ability  to  provide  insight 
into  the  implementation  problem.  Further  progress  in 
implementation  research  will  require  more  complex, 
realistic  models  and  the  development  of  alternative 
perspectives  for  viewing  implementation.  (41:xi) 

The  need  for  a  new  direction  in  future  research  efforts 
is  shared  by  several  researchers.  In  "A  Reassessment  of 
Implementation  Process  Models,"  Ananth  Srinivasan  and  Joseph 
Davis  conclude  that  both  the  wide  diversity  of  users  and  the 
rapid  changes  in  OR/MS  and  management  information  systems 
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technologies  has  "eroded"  the  relevance  of  efforts  to  model 
the  implementation  process  (47:64).  They  note  that  since 
quantitative  management  techniques  can  be  implemented  on 
personal  computer  systems,  the  traditional  models  of 
implementation  which  typically  involved  large  main-frame 
systems,  are  no  longer  useful  in  describing  the  new  decision- 
support  environment  (47:69). 

In  "Beyond  the  Model,"  Geoff  Lockett  also  notes  that 
implementation  research  itself  "is  in  a  state  of  transition" 
as  micro-computers,  artificial  intelligence,  and  management 
information  systems  become  increasingly  more  prevalent  in 
organizations  (24:174).  Lockett  notes  that  "managers  live  in 
an  environment  which  is  unusually  turbulent.  Much  of  their 
work  is  spent  fire-fighting  with  little  time  for  calm 
decision  making"  (24:175). 

Lockett  maintains  that  the  traditional  role  of  the 
systems  analyst  "has  become  blurred"  as  a  result  of  rapidly 
changing  OR/MS  and  management  information  system  technol¬ 
ogies,  and  he  recommends  that  future  implementation  research 
efforts  consider  this  (24:175). 

This  section  has  provided  an  overview  of  some  of  the 
major  implementation  research  efforts  of  the  past  15  years. 
The  remainder  of  this  literature  review  will  focus  specifi¬ 
cally  on  the  potential  applications  of  formal  R&D  project 
selection  techniques  as  a  means  to  enhance  the  effectiveness 
of  R&D  project  selection  and  resource  allocation  decisions. 
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The  Implementation  of  R&D  Project  Selection  Techniques 


In  '’The  Practice  of  Management  Science  in  R&D  Manage¬ 
ment,  "  Matthew  Libera tore  describes  the  R&D  project  selection 
process  as 

the  allocation  of  scarce  organizational  resources, 
namely,  funds,  manpower,  and  facilities,  to  a  set  of 
proposals  for  scientific  and  engineering  activities. 
(22:12) 

Consequently,  one  of  the  most  important  tasks  facing  the  R&D 

manager  is  to  select  the  most  promising  research  projects  in 

order  to  use  resources  effectively. 

Although  technological  developments  have  advanced 

tremendously  in  the  past  50  years,  the  process  by  which  R&D 

projects  are  selected  has  not.  Lowell  Steele  compared 

today's  project  selection  process  to  that  of  the  1930s,  and 

made  the  following  observation: 

Examining  the  management  of  industrial  R&D  over  the 
past  50  years  leaves  an  intriguing  impression:  a  member 
of  a  laboratory  of  the  1930s  would  feel  quite  comfort¬ 
able  with  the  way  a  laboratory  of  the  1980s  defines  its 
purposes  and  decides  what  programs  to  pursue.  He  would 
be  surprised  at  the  wealth  of  information  that  has  to 
be  evaluated,  at  the  number  of  people  involved  in  the 
process,  and  at  much  of  the  management  terminology. 

But  most  important,  he  would  not  be  surprised  at  the 
extent  to  which  informed  managerial  judgment  still 
plays  the  decisive  role  in  selecting  programs *  (48:21) 

In  the  forward  to  Management  of  Research  and  Innovation. 

Jordan  Baruch  notes  that  until  now,  most  R&D  managers  felt 

that  technological  innovation  "was  something  that  befell  a 

firm,  not  something  for  which  one  planned"  (6:vii) .  An 

examination  of  the  R&D  environment  provides  some  insight  into 

this  problem. 
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According  to  James  Matheson,  "uncertainty,  complexity, 
and  long  lead  times  are  fundamental  characteristics  of 
research  and  development"  (29:329).  Steele  substantiates 
this  opinion  with  his  description  of  R&D  management  in  the 
1980s.  First,  he  notes  that  there  is  a  "great  diversity  in 
management  practice"  (48:23).  In  spite  of  the  emergence  of 
numerous  quantitative  project  selection  techniques,  no  one 
technique  is  widely  accepted  as  the  standard.  Second,  Steele 
notes  an  ambivalence  on  the  part  of  managers  toward  the  use 
of  formal  project  selection  techniques.  In  describing  the 
effect  that  formal  techniques  have  on  the  R&D  project 
selection  process,  Steele  concludes  that  "they  might  as  well 
have  not  been  developed  (48:23).  Steele  also  maintains  that 
while  scientists  and  engineers  tend  to  seek  quantitative 
approaches  to  problems 

their  appreciation  of  the  complexities  and  subtleties 
of  project  selection  and  of  the  severe  limitations  on 
the  utility  of  those  techniques  that  have  been  developed 
make  managers  exceedingly  reluctant  to  give  them  a 
dominant  role  in  the  process.  (48:21) 

Despite  the  limited  use  of  formal  project  selection 

techniques,  Steele  notes  that  close  attention  is  given  to 

the  process  of  R&D  project  selection  (48:23).  This  is 

primarily  because  of  the  "increasingly  stringent  demands  for 

performance"  facing  today's  R&D  manager  (48:20). 

Despite  increasing  demands  for  more  effective  use  of 

resources,  most  R&D  managers  select  their  research  projects 

through  an  organizational  review  process: 
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Typically  these  reviews  include  an  examination  of 
progress  in  the  recent  past;  a  delineation  of  the  most 
important  problems  to  be  addressed  in  the  next  phase; 
consideration  of  the  human,  financial,  and  other 
resource  requirements  needed  for  optimum  progress; 
analysis  of  the  competitive  and  patent  status  of  the 
program;  evaluation  of  its  potential  benefit  and  the 
importance  of  timing;  and  finally,  consideration  of  the 
capability  and  desire  of  the  company  to  commercialize 
the  development.  (48:28) 

Steele  concludes  his  article  with  several  observations. 

First,  despite  the  limited  impact  that  quantitative 
techniques  have  had  on  the  project  selection  process,  the 
increasing  scale  and  complexity  of  the  R&D  environment  has 
forced  managers  to  seek  out  formal  procedures  for  project 
selection. 

Second,  quantitative  projection  techniques  "evoke  a 
love-hate  attitude  among  researchers"  (48:33).  Steele  notes 
that  "intellectual  bent  encourages  their  use,  while  wariness 
of  their  limitations  fosters  caution"  (48:33). 

Third,  Steele  notes  that  "the  demands  on  managerial  time 
and  limitations  on  availability  of  quantified  inputs"  are  the 
primary  reasons  for  the  limited  use  of  project  selection 
techniques  (48:34). 

Finally,  Steele  feels  that  "managerial  judgment  is  in 
no  danger  of  being  replaced  as  the  ultimate  umpire"  in 
project  selection  decisions  (48:34). 

Barriers  to  Implementation.  Several  researchers  have 
sought  to  determine  the  reasons  that  MS/OR  techniques  are 
often  resisted  by  managers. 
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Ivan  Vertinsky  suggests  that  the  limited  use  of  MS/OR 
techniques  is  due  to  1)  the  perception  among  managers  that 
MS/OR  is  a  threat  to  their  security;  2)  the  misunderstanding 
of  the  capabilities  of  MS/OR,  especially  among  older 
personnel;  and  3)  the  reliance  on  "traditional,  crisis- 
oriented  management"  (49:255). 

C.  Jackson  Grayson  notes  that  managers  often  reject 
MS/OR  techniques  due  to  1)  shortage  of  managerial  time;  2) 
inaccessibility  of  input  data;  3)  resistance  to  change,  4) 
the  slow  response  time  of  many  MS/OR  models,  and  5)  over¬ 
simplified  models  (13:25-27). 

In  a  more  detailed  examination  of  R&D  project  selection 
techniques,  E.  P.  Winkofsky  concluded  that  many  of  these 
techniques  possess  several  limitations.  Winkofsky' s 
conclusions  are  summarized  in  Table  6. 

In  "A  System  for  Using  R&D  Project  Evaluation  Methods," 
Souder  attempts  to  isolate  the  reasons  why  quantitative 
project  selection  techniques  are  often  ignored  by  R&D  man¬ 
agers.  He  maintains  that  many  of  the  older  quantitative 
techniques  are  not  widely  accepted  because  of  their  failure 
to  accurately  reflect  the  project  selection  process.  Souder 
found  that  many  quantitative  models  consider  only  the 
analytical  aspects  of  project  selection,  and  fail  to  con¬ 
sider  the  impact  that  "organizational  and  human  behavior" 
have  on  the  project  selection  process.  He  therefore  suggests 
that  before  a  quantitative  technique  can  be  useful  to  an  R&D 
manager,  organizational  goals  and  objectives  must  be  agreed 
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Table  6. 


Winkofsky's  Limitations  of  R&D  Project  Selection  Techniques 


1.  Inadequate  treatment  of  multiple,  often  interrelated 
criteria 

2.  Inadequate  treatment  of  project  interrelationships 
with  respect  to  value  contribution  and  to  resource 
utilization 

3.  No  explicit  recognition  and  incorporation  of  the 
experience  and  knowledge  of  the  R&D  manager 

4.  The  inability  to  recognize  and  treat  non-monetary 
aspects  such  as  establishing  and  maintaining 
balance  in  the  R&D  program 

5.  Perceptions  held  by  the  R&D  managers  that  the  models 
are  unnecessarily  difficult  to  understand  and  use 

6.  Inadequate  treatment  of  the  time  variant  property  of 
data  and  selection  criteria 


Adapted  from  (50:191) 

upon,  there  must  be  a  feeling  of  trust  among  workers,  and 
all  involved  parties  must  have  an  in-depth  knowledge  of  the 
projects  being  considered  (43:33-34). 

The  focus  on  organizational  dynamics  and  differences  by 
many  implementation  researchers  prompted  a  comparison  be¬ 
tween  R&D  in  the  public  and  private  sectors.  The  following 
section  explores  the  similarities  and  differences  between 
R&D  management  in  these  two  environments. 

Public  and  Private  Sector  R&D.  The  literature  comparing 
R&D  in  the  public  and  private  sectors  indicates  that  there 
may  be  significant  differences  between  R&D  project  selection 
in  the  public  and  private  sectors,  and  that  this  may  have  an 
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effect  on  the  applicability  of  quantitative  project  selection 
techniques. 

In  "The  Multiple  Functions  of  Formal  Aids  to  Decision 

Making  in  Public  Agencies,"  J.  David  Roessner  noted  that 

"R&D  project  selection  decisions  in  public  agencies  must 

incorporate  a  much  wider  range  of  criteria  than  equivalent 

decisions  in  private  firms"  (36:128).  However,  Roessner 

suggests  that  quantitative  techniques  can  be  effective  in 

public  sector  project  selection  to  "screen  candidate 

projects,  to  help  structure  the  allocation  problem,  and  to 

help  make  the  allocation  project  more  systematic"  (36:127). 

Frederick  Betz  also  examined  the  differences  between  R&D 

in  the  public  and  private  sectors,  concluding  that: 

In  the  process  of  contracting  out  R&D,  the  manager  in 
government  is  constrained  in  the  project  selection 
process  by  political  and  participatory  considerations 
and  legalistic  regulation.  (2:236) 

Because  a  significant  portion  of  government  R&D  is  performed 

under  contract  (up  to  71%  in  the  DoD) ,  Betz  maintains  that 

"only  in  managing  intramural  federal  R&D  does  the  public 

sector  R&D  manager  closely  resemble  his  private  sector 

counterpart"  (2:236).  Betz  also  notes  that 

as  government  research  moves  in  the  R&D  spectrum  from 
basic  and  applied  research  to  development,  then  the 
governmental  R&D  manager's  selection  of  projects  begins 
to  look  more  similar  to  the  industrial  manager's  con¬ 
cerns.  (2:236) 

Radnor  found  that  government  organizations  generally 
have  less  well-defined  outputs  and  goals  than  does  business 
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and  thus  Mwe  could  then  expect  implementation  problems  of  a 
differing  order  of  magnitude'1  (35:977)  . 

K.  G.  Feller  reviewed  methods  for  evaluating  R&D  project 
selection  techniques  for  the  U.S.  Department  of  Energy's 
Office  of  Advanced  Conservation  Technologies.  He  found  that 
the  assumptions  on  which  many  project  selection  models  are 
based  are  often  inappropriate  when  applied  to  government  R&D 
programs.  Speaking  of  formal  project  selection  methods. 
Feller  maintains  that 

1)  The  methods  imply  centralized  planning  and  cen¬ 
tralized  authority  to  take  the  necessary  actions 
over  the  time  needed  to  develop  and  implement  a 
technology; 

2)  The  methods  imply  an  easily  agreed-upon  measure  of 
merit ;  and 

3)  The  methods  imply  a  readily-available  source  of 
high-quality  information  about  the  R&D  objects,  the 
application  of  the  subject  technology,  and  the 
market  for  the  product  being  developed.  (10:4) 

There  appears  to  be  substantial  differences  in  the 
management  of  R&D  in  the  public  and  private  sectors.  Bearing 
in  mind  these  differences,  this  research  will  investigate  the 
R&D  project  selection  process  at  the  Air  Force  Wright 
Aeronautical  Laboratories,  in  a  effort  to  understand  the 
organizational  climate  and  the  applicability  of  formal 
project  selection  techniques. 

R&D  Project  Selection  Techniques 

This  final  section  examines  the  range  of  R&D  selection 
models  available  to  laboratory  managers. 
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A  formal  R&D  project  selection  technique  is  a  quan¬ 
titative  method  used  to  assign  values  to  individual  R&D 
projects  or  groups  of  projects  (36:124).  Models  which  are 
used  to  evaluate  individual  research  projects  are  known  as 
project  selection  models  (15:16).  Models  which  are  used  to 
evaluate  a  combination  of  interrelated  research  efforts  are 
known  as  portfolio  selection  models  (16:21).  It  is  estimated 
that  "literally  thousands  of  project  and  portfolio  selection 
models  are  available"  (46:36).  Fortunately,  there  is  a 
consensus  among  researchers  that  these  models  can  be  grouped 
into  four  categories:  1)  scoring  models;  2)  economic  models; 
3)  constrained  optimization  models;  and  4)  decision  theory 
models  (4:41;  12:292;  21:21;  46:36). 

This  review  will  provide  the  reader  with  a  concise 
description  of  each  of  the  categories  of  models,  including 
their  uses,  strengths,  and  weaknesses. 

Scoring  Models.  Of  the  four  categories  listed  above, 
scoring  models  are  considered  to  be  the  least  complex  and 
easiest  to  use  (12:292).  Scoring  models  are  used  to  obtain  a 
relative  ranking  of  competing  research  projects.  Application 
of  a  scoring  model  begins  with  the  development  of  standards 
against  which  each  research  project  will  be  measured.  Each 
project  under  consideration  is  then  rated  numerically  as  to 
how  well  it  meets  each  of  the  standards.  The  individual 
scores  are  then  summed,  providing  a  relative  ranking  of  the 
projects  (21:22).  Because  scoring  models  can  incorporate 
quantitative  input  data,  they  are  often  used  to  select 
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projects  in  the  initial  phases  of  research,  when  accurate 
cost  and  benefit  data  is  difficult  to  generate  (36:126). 


Scoring  models  are  often  selected  for  their  simplicity, 
ease  of  use,  and  flexibility.  Measurement  standards  are 
easily  developed,  and  do  not  have  to  conform  to  the  strict 
input  specifications  that  are  typical  of  many  other  selec¬ 
tion  models  (15:16). 

In  general,  scoring  models  have  two  shortcomings. 

First,  they  provide  only  a  relative  ranking  of  competing 
projects.  Scoring  models  are  not  designed  to  provide 
quantitative  data,  such  as  development  costs  or  expected 
benefits.  Second,  scoring  models  are  relatively  informal, 
and  lack  the  mathematical  basis  of  many  project  selection 
techniques.  Consequently,  their  results  are  often  difficult 


to  justify  (15:18). 


Economic  models  are  used  to  rank 


projects  based  on  their  potential  to  provide  financial 
benefits  over  time.  A  number  of  economic  techniques  such  as 
"sales-to-benefit  ratio,  cash  flow  payback,  net  present 
value,  benefit-to-cost  ratio  and  internal  rate  of  return" 


are  used  by  these  models  (8:30).  Because  economic  models 
rely  heavily  on  cost  and  benefit  data,  they  are  most 
applicable  during  the  later  stages  of  a  research  effort, 
when  the  project  has  stabilized  and  accurate  cost  estimates 
can  be  made  (21:21). 

Economic  models  have  two  drawbacks.  First,  while 
projections  of  development  costs  and  potential  benefits  are 
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very  useful,  accurate  estimates  are  not  easily  obtained  for 
projects  in  the  earliest  phases  of  research.  Thus,  economic 
models  are  not  appropriate  for  project  selection  during  this 
time.  Second,  economic  models  focus  only  on  the  financial 
benefits  of  a  research  project.  Other  possible  selection 
factors,  such  as  technical  leadership  and  research  prestige, 
are  not  considered  (19:172). 

Constrained  Optimization  Models.  Constrained  optimi¬ 
zation  models  are  used  to  select  projects  which  are  subject 
to  resource  constraints.  Resource  constraints  typically 
involve  funding  or  manpower  limitations  on  the  part  of  the 
laboratory.  Constrained  optimization  models  are  based  on 
mathematical  optimization  techniques  such  as  linear,  non¬ 
linear,  integer  and  dynamic  programming  (16:21).  This 
category  of  models  is  most  suited  for  large  scale  portfolio 
selection  situations  involving  numerous  interrelated  projects 
(16:22;  46:38). 

Although  constrained  optimization  models  are  solidly 
grounded  in  MS/OR  theory,  the  literature  indicates  that  the 
use  of  these  models  is  especially  limited  in  R&D  (12:295; 
21:22;  46:39).  According  to  Dr.  William  Souder  and  Tomislav 
Mandakovic , 

few  real  situations  are  linear,  and  forcing  them 
into  a  linear  programming  model  may  result  in  a 
meaningless  solution.  To  avoid  such  difficulties, 
a  variety  of  integer  and  nonlinear  programming 
models  have  been  devised.  However,  the  models  are 
difficult  to  solve  and  the  required  input  data  is 
not  easy  to  obtain  with  any  degree  of  precision  or 
confidence.  (46:38) 
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Decision  Theory  Models.  Decision  theory  models  are 
used  to  predict  the  overall  success  of  a  sequence  of  inter¬ 
related  projects  over  a  period  of  time  (16:19).  This 
category  is  comprised  of  "decision  tree,  decision  theory, 
utility  theory,  fault  tree  and  relevance  tree  models"  (46:- 
39)  . 

The  ability  of  decision  theory  models  to  consider  a 
range  of  uncertainty  is  well  documented  as  their  major 
strength  (1:152;  16:19;  17:29).  In  addition,  decision  theory 
models  force  a  manager  to  consider  the  importance  of  each 
interrelated  sub-project  to  the  outcome  of  the  overall 
project  (16:20). 

According  to  the  literature,  decision  theory  models 
have  two  major  weaknesses.  As  with  the  constrained  opt¬ 
imization  models,  the  first  drawback  focuses  on  the  amount 
and  quality  of  input  data  required.  These  requirements 
include  estimates  of  both  the  cost  and  benefit  of  each  sub- 
project,  as  well  as  an  estimate  of  the  probability  distribu¬ 
tion  for  each  project  outcome  (16:20).  The  second  drawback 
concerns  the  complexity  of  the  model  itself.  Farrell  Jenson 
and  John  Lawson,  authors  of  "Quantifying  R&D  Projects  in  the 
Face  of  Uncertainty"  note  that 

the  major  problem  which  will  be  faced  in  imple¬ 
menting  this  technique  is  building  the  confidence 
of  management.  Oftentimes,  a  good  technique  will 
not  be  implemented  simply  because  it  is  not  under¬ 
stood.  (17:29) 
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Chapter  Summary 


This  literature  review  began  with  an  overview  of  the 
research  devoted  to  the  implementation  of  MS/OR  techniques 
as  a  management  tool.  It  examined  the  implementation  gap  - 
the  gap  between  MS/OR  theory  and  application  -  and  inves¬ 
tigated  several  variables  which  have  been  found  to  influence 
the  implementation  of  MS/OR  techniques  in  organizations. 

Three  "generations'*  of  implementation  research  were  dis¬ 
cussed.  Those  research  efforts  that  are  particularly 
applicable  to  this  effort  were  discussed  in  detail. 

The  second  section  focused  specifically  on  the  imple¬ 
mentation  of  R&D  project  selection  techniques  in  organiza¬ 
tions.  First,  the  R&D  environment  was  discussed.  Next,  some 
of  the  common  barriers  which  have  precluded  the  use  of 
quantitative  techniques  were  examined.  Finally  a  comparison 
was  made  between  the  demands  on  the  R&D  manager  in  government 
and  in  private  industry. 

In  the  final  section,  four  categories  of  quantitative 
project  selection  techniques  were  discussed,  including  their 
strengths  and  weaknesses.  This  literature  review  sets  the 
stage  for  an  examination  of  the  R&D  project  selection 
process  at  the  Air  Force  Wright  Aeronautical  Laboratories. 
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As  noted  in  Chapter  I,  this  research  investigated  three 
facets  of  the  R&D  project  selection  process  at  AFWAL.  They 


consisted  of  1)  the  methods  used  by  managers  to  select  R&D 
projects  and  allocate  resources;  2)  the  attitudes  and 
perceptions  of  managers  toward  the  use  of  formal  R&D  project 
selection  techniques;  and  3)  the  factors  affecting  the 
adoption  of  these  techniques  by  R&D  managers. 

To  accomplish  this  task,  surveys  were  administered  to  a 
non-random  sample  of  43  AFWAL  managers.  In  addition,  a 
subset  of  12  managers  were  selected  for  follow-up  interviews. 
The  selection  of  respondents,  survey  instrument,  statistical 
analyses,  and  interview  process  are  described  below. 

selection  gf  MsRQrwtent? 

To  adequately  investigate  the  R&D  project  selection  at 
AFWAL,  it  was  necessary  to  seek  out  those  managers  who 
participated  in  the  decision  making  process.  The  results  of 
a  1985  AFIT  thesis  by  Captain  Jeremy  Prince  provided 
information  leading  to  the  identification  of  respondents. 

In  his  research,  Prince  found  that  "the  final  decisions  on 
project  selection  are  often  reserved  for  the  Branch  Chief 
level  or  higher"  (31:40).  Consequently,  a  non-random  sample 
of  respondents  occupying  positions  c  f  Deputy  Division  Chief 
or  higher  was  selected.  Surveys  were  sent  to  all  appro- 


priate  Division  Chiefs,  Chief  Scientists,  Deputy  Directors 
and  Directors.  Surveys  were  not  sent  to  Division  Chiefs  of 
staff  organizations,  however,  the  Director  of  Plans,  the 
Director  of  Programs,  and  the  Director  of  Management 
Information  Systems  were  included. 

Interviews  were  conducted  with  a  subset  of  those  who 
were  surveyed.  The  interview  candidates  were  chosen  based  on 
their  response  to  survey  question  17  (Appendix  A) .  Only 
those  who  perceived  that  they  had  a  strong  influence  on  the 
project  selection  process  were  selected.  In  all,  12 
interviews  were  conducted.  To  fully  represent  a  cross- 
section  of  AFWAL  managers,  three  interviews  were  conducted 
at  each  of  the  four  laboratories. 

Survey  Instrument 

In  "Implementation  and  Organizational  Validity:  An 

Empirical  Investigation,"  Schultz  and  Slevin  note  that 

one  way  to  stimulate  the  orderly  collection  of  data 
across  a  field  of  research  as  broad  as  implementation 
is  to  provide  a  meaningful  and  easily  used  instrument 
for  data  collection.  (38:154) 

The  survey  used  this  research  was  adapted  from  a 
Likert-based  instrument  provided  by  Dr.  William  Souder,  a 
Professor  at  the  University  of  Pittsburgh.  An  overview  of 
Souder* s  research  approach  is  included  in  Chapter  II  of  this 
document.  A  more  detailed  account  is  contained  in  "Methodol¬ 
ogy  for  Increasing  the  Adoption  of  R&D  Project  Selection 
Models,"  and  in  "An  Organizational  Intervention  Approach  to 
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the  Design  and  Implementation  of  R&D  Project  Selection 
Models"  (44;  45). 

The  survey  instrument  used  in  this  research  is  contained 
in  Appendix  A.  Because  the  instrument  was  developed  and 
used  by  an  expert  in  the  field  of  implementation  research, 
its  internal  validity  was  not  questioned  (9:99) . 

Minor  wording  changes  were  made  to  Souder's  instrument 
to  make  it  directly  applicable  to  AFWAL.  In  addition,  the 
data  from  the  Likert-based  questions  was  analyzed  at  the 
interval  level  to  take  advantage  of  parametric  statistical 
techniques.  These  modifications  did  not  change  the  original 
intent  of  the  survey  instrument. 

In  his  research,  Souder  utilized  a  longitudinal  design 
to  identify  an  originating  set  of  variables  that  affect  the 
adoption  of  R&D  project  selection  techniques.  The  stability 
of  the  original  set  of  variables  was  established  by  admin¬ 
istering  a  second  survey  to  the  respondents  following  a 
training  session  in  which  formal  R&D  project  selection 
techniques  were  discussed  (45:74).  Due  to  time  limitations, 
this  research  focused  on  deriving  only  the  initial  set  of 
adoption-affecting  variables.  Suggestions  for  a  follow-on 
longitudinal  effort  are  described  in  Chapter  V. 

The  survey  used  in  this  research  effort  was  divided 
into  three  parts.  Each  part  is  described  below. 

Part  1.  The  responses  to  survey  questions  1  through  16 
were  used  to  investigate  Research  Question  1.  These  survey 
questions  were  designed  to  measure  levels  of  familiarity  and 
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usage  of  a  variety  of  R&D  project  selection  techniques,  as 
well  as  to  determine  the  extent  to  which  R&O  project 
selection  techniques  are  used  at  AFWAL. 

Questions  17  through  28  were  designed  to  measure  the 
respondent's  attitudes  and  perceptions  toward  the  use  of 
formal  project  selection  techniques.  These  questions  provide 
the  basis  for  the  response  to  Research  Question  2. 

Part  II.  Part  II  of  the  survey  was  used  to  identify 
the  factors  affecting  the  adoption  of  R&D  project  selection 
techniques  in  response  to  Research  Question  3.  This  portion 
of  the  survey  was  used  to  derive  a  set  of  adoption-affecting 
variables.  The  variables  that  were  tested  in  this  research 
were  derived  from  the  works  of  Dr.  Souder,  and  are  defined  in 
Table  7  (44 ;  45) . 

Thirty  survey  questions  were  used  to  investigate  the 
variables  shown  in  Table  7.  At  least  two  questions  were 
used  to  investigate  each  variable.  The  survey  responses  to 
questions  designed  around  a  particular  variable  were 
aggregated  and  averaged  over  the  number  of  questions  used. 

In  some  instances  it  was  necessary  to  invert  the  Likert 
scale  to  be  consistent  with  the  remainder  of  the  survey. 

This  recoding  process  accounts  for  the  all-positive  corre¬ 
lation  matrix  shown  in  Chapter  IV. 

Part  III.  Part  III  was  used  to  gather  demographic  data 
about  the  respondents.  In  addition  to  demographic  data,  Part 
III  of  the  survey  provides  information  such  as  the  budgetary 
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Table  7. 


Variable  Definitions 


1.  Communicability .  The  ease  with  which  the  subject 
perceives  that  project  selection  techniques  can  be 
communicated  to  others. 

2.  Appropriateness  of  Outputs ♦  The  degree  to  which  the 
subject  perceives  that  a  technique's  output  data  is 
relevant. 

3.  Appropriateness  of  Mathematics.  The  degree  to  which 
the  subject  perceives  that  the  mathematical  basis 
for  a  technique  is  relevant. 

4.  Availability  of  Inputs.  The  perceived  ease  with 
which  input  data  is  obtained  for  a  technique. 

5.  Upward  Compatibility.  The  degree  of  perceived 
compatibility  with  superiors  if  project  selection 
techniques  are  used  in  the  decision  process. 

6.  Downward  Compatibility.  The  degree  of  perceived 
compatibility  with  subordinates  if  project  selection 
techniques  are  used  in  the  decision  process. 

7.  Process?,  Compatibility-  The  perceived  compatibility 
of  existing  project  selection  processes  and  formal 
project  selection  techniques. 

8.  Uncertainty  Absorption.  The  subject's  perception  of 
the  ability  of  a  project  selection  technique  to 
reduce  the  amount  of  uncertainty  in  decisions. 

9.  Information  Value.  The  subject's  perception  of  the 
value  of  both  the  amount  and  type  of  information 
provided  by  the  model. 

10.  Process  Value.  The  perceived  need  for  a  formal 
structure  in  the  subjects  own  decision-making 

11.  Willingness  to  Adopt.  The  subject's  willingness  to 
adopt  a  formal  R&D  project  selection  technique 

for  routine  and  frequent  use  over  a  long  period  of 
time. 


Adapted  from  (44;  45) 


responsibilities  of  the  respondents,  and  their  willingness  to 
attend  a  class  or  seminar  on  R&D  project  selection  tech¬ 
niques. 

S  ta  t  i  ati  gal-Ana  ly.sg  § 

Several  statistical  techniques  were  used  to  analyze  the 
survey  data  including:  1)  analysis  of  variance;  2)  relia¬ 
bility  analysis;  3)  correlation  analysis;  4)  factor  analysis; 
and  5)  descriptive  statistics.  The  survey  data  was  processed 
on  AFIT's  Academic  Support  Computer  using  the  "Statistical 
Package  for  the  Social  Sciences  -  X"  (SPSSx) .  The  statis¬ 
tical  analyses  used  in  this  research  are  described  below. 

The  results  of  this  data  analysis  are  presented  in  Chapter 
IV,  and  are  discussed  in  Chapter  V. 

Analysis  of  Variance.  A  Kruskal-Wallis  non-parametric 
analysis  of  variance  was  used  to  identify  statistically 
significant  differences  between  the  responses  of  the  managers 
at  each  of  the  four  AFWAL  laboratories.  The  Kruskal-Wallis 
test  was  selected  for  its  robustness  and  conservatism 
(7:622) . 

The  test  was  applied  to  survey  questions  1  through  15 
to  identify  statistically  significant  differences  of  the 
levels  of  familiarity  and  usage  of  R&D  project  selection 
techniques  between  the  managers  at  each  of  the  four  AFWAL 
laboratories.  The  Kruskal-Wallis  test  was  also  used  to 
identify  differences  in  opinions  among  the  managers  in  each 
lab  (survey  questions  17  through  27) . 
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In  each  case,  it  was  hypothesized  that  the  responses 
from  each  laboratory  would  be  identical.  The  Kruskal-Wallis 


tests  were  conducted  a  .05  level  of  significance. 

Reliability.  Correlation,  and  Factor  Analysis.  To 

identify  factors  affecting  the  adoption  of  R&D  project 

selection  techniques,  a  three-part  approach  was  used.  First, 

the  internal  reliability  of  the  questions  in  Part  II  of  the 

survey  were  checked  using  Cronbach's  Alpha.  In  cases  where 

Cronbach's  Alpha  could  be  increased  by  at  least  .1,  the 

survey  question  was  considered  for  elimination.  With  this 

criteria,  survey  question  29  was  eliminated.  Next,  an  11  by 

11  Pearson  correlation  matrix  was  derived  by  aggregating 

survey  responses  for  each  variable.  All  variables  which  were 

found  to  correlate  with  the  variable  "willingness  to  adopt" 

at  a  level  of  significance  of  .05  were  considered  to 

influence  the  adoption  of  R&D  project  selection  techniques, 

and  were  included  in  the  factor  analysis. 

In  the  final  step,  factor  analysis  was  used  to  reduce 

the  adoption-affecting  variable  set  into  factors. 

Factor  analysis  is  a  technique  for  removing  redundancy 
from  a  set  of  correlated  variables  and  representing  the 
variables  with  a  smaller  set  of  derived  variables,  or 
factors.  (18:378) 

In  the  principal  components  extracts,  all  factors  whose 
eigenvalue  was  greater  than  or  equal  to  1.0  were  selected. 
According  to  Sam  Kash  Kachigan,  "eigenvalues  correspond  to  the 
equivalent  number  of  variables  which  the  factor  represents" 
(18:387) . 


A  varimax  rotation  was  used  to  redistribute  the 
explained  variance  among  the  extracted  factors.  This 
iterative  technique  is  used  to  clarify  the  factor  loadings, 
and  aids  in  the  assignment  of  variables  to  factors  (18:389). 

Variables  were  assigned  to  factors  based  on  their 
loadings  after  the  varimax  rotation.  Kachigan  notes  that 
"the  analyst  must  use  personal  judgment  as  to  what  con¬ 
stitutes  a  meaningfully  high  loading"  (18:393).  Furthermore, 
he  notes  that  "loadings  of  .3,  .4,  or  .5  are  most  often  used 
as  lower  bounds"  (18:393).  For  conservatism,  variables  were 
assigned  to  a  factor  only  if  their  factor  loading  was 
greater  than  .5. 

The  final  step  in  the  factor  analysis  process  involved 

the  naming  of  the  factors.  Again,  Kachigan  notes  that  this 

step  involves  personal  judgment: 

Careless  or  casual  naming  of  the  factors  might  be 
completely  misleading,  jeopardizing  the  conclusions  of 
the  study.  It  is  a  fact  that  the  identity  of  the 
contributing  variables  is  often  forsaken  once  a  factor 
has  been  named,  and  the  label  is  then  communicated  to 
those  who  would  hope  to  apply  the  results  of  the 
research.  It  is  wise,  therefore,  to  always  study  the 
variables  which  define  a  factor.  (18:394) 

Because  the  stability  of  the  adoption-affecting 

variables  set  derived  from  this  research  has  not  been  confirmed 

by  follow-on  research,  the  advice  offered  by  Kachigan  must  be 

considered. 

When  utilizing  factor  analysis  techniques,  it  is 
necessary  to  consider  the  size  of  the  survey  sample  in 
relation  to  the  number  of  variables  being  tested.  To  measure 
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the  adequacy  of  the  sample,  the  Kaiser-Meyer-Olkin  measure  of 
sampling  adequacy  was  used.  This  technique  is  described  in 
Factor  Analysis:  Statistical  Methods  and  Practical  Issues 
(20:53).  It  should  be  noted  that  when  investigating  a  small 
group,  such  as  the  R&D  managers  at  AFWAL,  sampling  adequacy 
can  be  jeopardized  if  a  larger  group  of  adoption-affecting 
variables  are  tested.  This  must  be  considered  in  any  follow- 
on  efforts. 

Descriptive  Statistics.  Frequency  responses  were  used 
to  summarize  and  present  the  data  from  Parts  I  and  III  of 
the  survey.  Inferences  about  management's  use  of  and  familiar¬ 
ity  of  R&D  project  selection  techniques,  as  well  as  their 
perceptions  of  the  utility  of  these  techniques,  were  drawn 
from  these  frequency  distributions,  as  well  as  from  the  data 
collected  in  the  interviews. 

Interview  Process 

As  noted  in  Chapter  II,  intuition  and  expert  judgment 
often  play  a  significant  role  in  R&D  project  selection 
(48:28).  To  fully  investigate  this,  follow-on  interviews 
were  conducted  to  supplement  the  survey  instrument.  In  Busi¬ 
ness  Research  Methods.  C.  William  Emory  notes  that  the  inter¬ 
view  process  provides  an  excellent  method  for  extracting  detailed 
information.  However,  it  is  also  noted  that  interviews  are 
often  costly  to  conduct  because  of  travel  costs  to  and  from 
the  interview  location  (9:161).  Furthermore,  busy  executives 
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are  often  hesitant  to  grant  long  interviews  due  to  constraints 
on  their  time. 

Nonetheless,  because  of  the  proximity  of  AFIT  to  the 
laboratories,  a  series  of  semi-structured  interviews  were 
conducted,  both  in  person  and  via  telephone.  Appendix  C 
contains  a  list  of  the  interview  questions.  Candidates  were 
initially  contacted  and  interviewed  by  telephone.  After 
receiving  their  consent,  an  interview  was  conducted  by 
telephone,  unless  a  personal  interview  was  requested.  Three 
of  the  12  interviews  were  conducted  in  person.  The  interview 
questions  contained  in  Appendix  C  were  used  only  as  a  guide. 
The  respondents  were  encouraged  to  discuss  any  thoughts  they 
had  concerning  the  R&D  project  selection  process. 

Chapter  Summary 

This  chapter  provided  an  overview  of  the  methodology 
used  in  this  research  effort.  First,  the  basis  for  selecting 
the  non-random  sample  of  AFWAL  managers  was  presented.  Next, 
the  survey  instrument  and  the  statistical  techniques  used  to 
process  the  survey  data  were  discussed.  Finally,  the 
personal  and  telephone  interview  process  was  discussed. 

In  the  following  chapter,  the  results  of  the  survey  data 
are  presented.  These  results  are  discussed  in  detail  in 
Chapter  V. 


This  chapter  provides  a  summary  presentation  of  the 
responses  to  the  survey  instrument.  These  responses  are 
discussed  and  analyzed  in  detail  in  Chapter  V. 

Of  the  43  surveys  administered,  32  were  completed  and 
returned  (74.4  percent).  Tabular  frequency  distributions 
are  presented  for  both  the  demographic  data  and  attitudinal 
measures.  In  some  instances,  column  percentages  do  not  sum 
to  100  percent  due  to  computer  rounding  or  missing  values. 

A  Kruskal-Wallis  analysis  of  variance  was  used  to  test 
for  statistically  significant  differences  between  the  four 
AFWAL  laboratories  in  familiarity  with,  use  of,  and  opinions 
about  R&D  project  selection  techniques.  The  identification 
of  adoption-affecting  variables  and  their  associated  factors 
was  accomplished  using  a  three-step  approach.  First,  the 
internal  reliability  of  the  survey  questions  in  Part  II  was 
assessed  using  Cronbach's  Alpha.  Next,  a  correlation  matrix 
was  developed  to  identify  associations  between  the  ten  test 
variables  and  the  variable  ’'willingness  to  adopt."  Finally, 
factor  analysis  was  performed,  utilizing  a  varimax  rotation, 
to  reduce  the  initial  adoption-affecting  variables  set  into 
factors . 

Demographic  Data 

Demographic  data  was  collected  on  all  respondents.  The 


demographic  data  includes  the  age,  rank,  sex,  education 


(highest  degree  obtained,  course-work  in  MS/OR  and  laboratory 
management,  and  the  respondent's  willingness  to  attend  a 
course  in  R&D  project  selection  techniques) ,  and  organiza¬ 
tional  affiliation  of  each  respondent.  All  respondents  were 
male.  Tables  8  through  12  provide  a  summary  of  the  demo¬ 
graphic  data. 

Table  8. 

Distribution  of  Respondents  by  Age 


Age  (years) 

Frequency 

Percent 

20-25 

0 

0.0 

26-30 

0 

0.0 

31-35 

0 

0.0 

36-40 

1 

3.1 

41-45 

4 

12.5 

46-50 

10 

31.3 

51-55 

9 

28.1 

56  + 

8 

25.0 

Total  32 

100.0 

Table  9. 


Distribution  of  Respondents  by  Organization 


Organization 

Frequency 

Percent 

Aero  Propulsion  Lab 

5 

15.6 

Avionics  Lab 

8 

25.0 

Flight  Dynamics  Lab 

9 

28.1 

Materials  Lab 

8 

25.0 

AFWAL  Staff  Organizations 

2 

6.3 

Total 

32 

100.0 
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Table  10. 


Distribution  of  Respondents  by  Highest  Educational  Degree 


Table  21. 


Distribution  of  Respondents  by  Applicable  Course-work 


Table  12. 

Respondent's  Willingness  to  Attend  a  Course 
In  R&D  Project  Selection  Techniques 


Yes 

Response 

No 

Frequency 

Percent 

Frequency 

Percent 

26 

80 

6 

20 
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The  information  contained  in  Table  13  summarizes  the 
results  of  survey  questions  1  through  15.  These  questions 
measure  the  respondent's  familiarity  with  and  use  of  several 
of  the  most  popular  R&D  project  selection  techniques. 

Survey  questions  17  through  27  were  analyzed  together  to 
measure  the  respondent's  attitudes  and  perceptions  toward  the 
utility  of  R&D  project  selection  techniques. 

Survey  question  17  examined  the  respondent's  perceived 
influence  on  his  organization's  project  selection  process. 
These  responses  were  used  to  assess  the  adequacy  of  the  non- 
random  sample.  Table  14  provides  a  summary  of  the  data. 

The  respondent's  perceptions  concerning  the  adequacy  of 
his  organization's  project  selection  process  was  investigated 
in  survey  question  18.  The  results  are  summarized  in  Table 
15. 

Survey  questions  19  and  20  were  designed  to  measure  the 
respondent's  attitudes  concerning  the  potential  worth  of  R&D 
project  selection  techniques,  both  as  a  personal  management 
tool,  and  as  a  tool  for  use  by  his  subordinates.  These 
responses  are  contained  in  Tables  16  and  17. 

A  series  of  four  questions  were  used  to  assess  the 
respondent's  overall  confidence  in  the  capabilities  of  R&D 
project  selection  techniques.  Question  21  sought  to 
determine  how  important  an  organizational  decision  to  use 
R&D  project  selection  techniques  on  all  R&D  project  evalu¬ 
ations  would  be  to  the  respondent.  Question  22  asked  the 
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Table  13 


Familiarity  with  and  Use  of  R&D  Project  Selection  Techniques 


Technique 

Not 

Familiar 

(Percent) 

Familiar 

(Percent) 

Use 

Regularly 

(Percent) 

Q-Sort 

96.9 

3.1 

0.0 

Checklists 

53.1 

31.3 

15.6 

Scoring 

Models 

62.5 

34.4 

3.1 

Delphi 

18.8 

65.6 

15.6 

Cost/Benefit 

Ratios 

25.0 

65.6 

9.4 

Payback 

Period 

46.9 

46.9 

6.3 

Net  Present 
Value/Intemal 
Rate  of  Return 

81.3 

15.6 

3.1 

Portfolio 

Models 

90.6 

9.4 

0.0 

Risk  Analysis/ 
Monte  Carlo 

43.8 

56.3 

0.0 

Decision  Trees/ 
Decision  Analysis 

25.0 

71.9 

3.1 

Linear 

Programming 

62.5 

37.5 

0.0 

Integer 

Programming 

90.6 

9.4 

0.0 

Nonlinear 

Programming 

84.4 

15.6 

0.0 

Dynamic 

Programming 

81.3 

18.8 

0.0 

Goal 

Programming 

87.5 

9.4 

3.1 

Table  14. 


Perceived  Influence  on  the  Project  Selection  Process 


Response 

Likert 

Scale 

Frequency 

Percent 

Very  Strong  Influence 

1 

6 

18.8 

2 

10 

31.3 

3 

5 

15.6 

Some  Influence 

4 

5 

15.6 

5 

3 

9.4 

6 

1 

3.1 

No  Influence 

7 

2 

6.3 

Total 

32 

100.0 

Table 

15. 

Evaluation  of  the  Current  Project  Selection 

Process 

Response 

Likert 

Scale 

Frequency 

Percent 

Needs  Great  Improvement 

1 

4 

12.5 

2 

2 

6.3 

3 

16 

50.0 

Uncertain 

4 

0 

0.0 

5 

4 

12.5 

6 

5 

15.6 

Needs  No  Improvement 

7 

1 

3.1 

Total 

32 

100.0 

respondent  to  indicate  his  preference  if  such  a  decision  were 
made.  Survey  questions  23  and  24  were  similar  to  questions 
21  and  22 ,  however,  in  questions  23  and  24,  the  organiza¬ 
tional  decision  concerned  the  use  of  R&D  project  selection 
techniques  on  half  of  their  project  selection  decisions. 
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Table  16. 


Evaluation  of  R&D  Project  Selection  Techniques 
as  a  Personal  Management  Tool 


Response 

Likert 

Scale 

Frequency 

Percent 

Extremely  Beneficial 

1 

1 

3.1 

2 

8 

25.0 

3 

13 

40.6 

No  Benefit/Uncertain 

4 

7 

21.9 

5 

1 

3.1 

6 

1 

3.1 

Extremely  Detrimental 

7 

1 

3.1 

Total 

32 

100.0 

Table  17. 


Evaluation  of  R&D  Project  Selection  Techniques 
as  a  Management  Tool  for  Subordinates 


Response 

Likert 

Scale 

Frequency 

Percent 

Extremely  Beneficial 

1 

3 

9.4 

2 

8 

25.0 

3 

11 

34.4 

No  Benefit/Uncertain 

4 

7 

21.9 

5 

2 

6.3 

6 

1 

3.1 

Extremely  Detrimental 

7 

0 

0.0 

Total 

32 

100.0 

Together,  these  four  questions  examined  the  levels  of 
conviction  and  confidence  that  managers  have  in  these 
techniques.  The  responses  are  summarized  in  Tables  18 
through  21. 
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Table  18. 


Perceived  Importance  of  a  Decision  to  Use 
R&D  Project  Selection  Techniques  on  All  Projects 


Response 

Likert 

Scale 

Frequency 

Percent 

Very  Important 

1 

4 

12.5 

2 

5 

15.6 

3 

6 

18.8 

Uncertain 

4 

6 

18.8 

5 

2 

6.3 

6 

3 

9.4 

Very  Unimportant 

7 

€ 

18.8 

Total 

32 

100.0 

Table  19. 


Recommendation  Concerning  the  Use  of 
R&D  Project  Selection  Techniques  on  All  Projects 


Response 

Likert 

Scale 

Frequency 

Percent 

Definitely  Use 

1 

2 

6.3 

2 

3 

9.4 

3 

7 

21.9 

Uncertain 

4 

8 

25.0 

5 

2 

6.3 

6 

4 

12.5 

Definitely  Do  Not  Use 

7 

6 

18.8 

Total 

32 

100.0 

Survey  question  25  was  used  to  measure  the  respondent's 
opinions  about  the  likelihood  of  developing  a  R&D  project 
selection  technique  that  would  be  beneficial  to  the  organi¬ 
zation.  The  results  are  summarized  in  Table  22. 
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Table  20. 


Perceived  Importance  of  a  Decision  to  Use  R&D  Project 
Selection  Techniques  on  Fifty  Percent  or  More  of  Projects 


Response 

Likert 

Scale 

Frequency 

Percent 

Very  Important 

1 

6 

18.8 

2 

2 

6.3 

3 

7 

21.9 

Uncertain 

4 

6 

18.8 

5 

3 

9.4 

6 

1 

3.1 

Very  Unimportant 

7 

7 

21.9 

Total 

32 

100.0 

Table  21. 


Recommendation  Concerning  the  Use  of  R&D  Project 
Selection  Techniques  on  Fifty  Percent  or  More  of  Projects 


Response 

Likert 

Scale 

Frequency 

Percent 

Definitely  Use 

1 

3 

9.4 

2 

4 

12.5 

3 

7 

21.9 

Uncertain 

4 

8 

25.0 

5 

1 

3.1 

6 

5 

15.6 

Definitely  Do  Not  Use 

7 

3 

9.4 

*  1  Missing  Value 

Total 

31* 

96.9  * 

Finally,  survey  questions  26  and  27  measured  the  extent 
to  which  the  respondent  felt  pressure  to  quantitatively 
justify  R&D  resource  allocations.  Table  23  summarizes  the 
respondents'  perceptions  concerning  the  amount  of  pressure 
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Table  22. 


Perceived  Likelihood  of  a  Beneficial 
R&D  Project  Selection  Technique  Being  Developed 


Response 

Likert 

Scale 

Frequency 

Percent 

Very  Likely 

1 

3 

9.4 

2 

4 

12.5 

3 

4 

12.5 

Uncertain 

4 

5 

15.6 

5 

4 

12.5 

6 

4 

21.9 

Very  Unlikely 

7 

7 

3.1 

*  1  Missing  Value 

Total 

31* 

96.9  * 

Table  23. 


Perceived  Pressure  Exerted  from  Within  AFWAL 
to  Quantitatively  Justify  R&D  Expenditures 


Response 

Likert 

Scale 

Frequency 

Percent 

Extreme  Pressure 

1 

1 

3.1 

2 

3 

9.4 

3 

6 

18.8 

Some  Pressure 

4 

9 

28.1 

5 

6 

18.8 

6 

6 

18.8 

No  Pressure 

7 

1 

3.1 

Total 

32 

100.0 

generated  from  within  AFWAL.  Table  24  summarizes  the 
respondents'  perceptions  concerning  the  amount  of  pressure 
being  exerted  from  outside  sources  (e.g.  the  Air  Force 


Table  24. 


Perceived  Pressure  Exerted  from  Outside  AFWAL 
to  Quantitatively  Justify  R&D  Expenditures 


Response 

Likert 

Scale 

Frequency 

Percent 

Extreme  Pressure 

1 

4 

12.5 

2 

5 

15.6 

3 

9 

28.1 

Some  Pressure 

4 

6 

18.8 

5 

2 

6.3 

6 

2 

6.3 

No  Pressure 

7 

4 

12.5 

Total 

32 

100.0 

Office  of  Scientific  Research,  Air  Force  Systems  Command, 
etc . ) . 

In  survey  question  28,  the  respondent  was  asked  to 
describe  any  positive  or  negative  experiences  he  has  had 
while  using,  or  attempting  to  use  a  formal  project  selection 
technique.  Responses  to  this  question  are  presented  in 
Appendix  C.  Numbers  were  assigned  to  each  narrative  only  to 
provide  a  method  for  referencing  these  statements  in  the 
following  chapter.  The  identities  of  the  respondents  are 
confidential. 

The  Kruskal-Wallis  analysis  of  variance  tests  showed  no 
significant  differences  in  familiarity  with,  use  of,  or 
attitudes  toward  project  selection  techniques  between  the 
four  AFWAL  laboratories  at  a  .05  level  of  significance. 
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Adoption-Affecting  Variables  and  Factors 

As  noted  in  Chapter  III,  a  three-step  process  was  used 
to  identify  a  set  of  adoption-affecting  variables  and  group 
them  into  factors.  The  results  of  this  process  are  presented 
here. 


Cronbach's  Alpha.  A  reliability  analysis  was  performed 
to  assess  the  internal  reliability  of  the  30  questions  in 
Part  II  of  the  survey  instrument.  As  noted  in  Chapter  III, 
the  survey  responses  designed  around  a  particular  variable 
were  aggregated  to  form  a  single  measure  of  that  variable. 
The  variables,  the  survey  questions  from  which  they  were 
derived,  and  the  resulting  values  of  Cronbach's  Alpha  are 
summarized  in  Table  25.  Based  on  the  results  of  this 
analysis,  survey  question  29  was  eliminated. 

Pearson  Correlation  Coefficients.  The  next  step  in  the 
process  was  to  identify  a  statistically  significant  set  of 
adoption-affecting  variables.  A  correlation  matrix  was 
generated  for  all  eleven  variables  (see  Table  7) .  Each 
variable  was  examined  for  its  degree  of  correlation  with  the 
variable  "willingness  to  adopt."  Of  the  ten  variables 
tested,  nine  were  found  to  correlate  with  "willingness  to 
adopt"  at  a  level  of  significance  of  .05.  The  variable 
"communicability"  did  not  correlate  significantly.  The  nine 
remaining  variables  were  used  as  the  input  for  the  factor 
analysis.  It  should  be  noted  that  several  variables  also 
exhibited  significant  interrelationships  (.05  level  of 
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Table  25. 


Reliability  Analysis 


Variable  Name 

Survey  Questions 

Cronbach's  Alpha 

Communicabil ity 

38, 

39,  52 

.7641 

Appropriateness 
of  Outputs 

46, 

54,  55 

.6893 

Appropriateness 
of  Mathematics 

47, 

56 

.4181 

Availability  of 
Inputs 

48, 

57 

.8538 

Upward 

Compatibility 

32, 

41 

.7033 

Downward 

Compatibility 

33, 

42 

.8078 

Process 

Compatibility 

30, 

37 

.5543 

Uncertainty 

Absorption 

34, 

44 

.8872 

Information 

Value 

31/ 

35,  36, 

43 

.7649 

Process  Value 

40, 

53,  58 

.7204 

Willingness 

45, 

49,  50, 

51 

.8702 

to  Adopt 

(29 

eliminated) 

significance)  with  the  other  adoption-affecting  variables. 
Table  26  shows  the  Pearson  correlations  between  the  vari¬ 
ables.  The  aggregation  and  recoding  procedures  described  in 
Chapter  III  account  for  the  positive  nature  of  the  corre¬ 
lation  matrix. 

Factor  Analysis.  Factor  analysis  was  used  to  reduce 
the  adoption-affecting  variables  set  into  factors.  Due  to 


57 


Table  26 
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*  .05  Level  of  Significance 

**  .01  Level  of  Significance 


the  relatively  large  number  of  variables,  there  was  concern 
about  the  adequacy  of  the  sample  size.  A  Kaiser-Meyer-Olkin 
measure  of  sampling  adequacy  was  calculated  and  found  to  be 
.8054.  This  value,  according  to  Jae-on  Kim  and  Charles 
Mueller,  can  be  considered  "meritorious."  (20:54). 

Two  factors  were  extracted,  and  account  for  66.7  percent 
of  the  variance  of  the  data.  A  varimax  rotation  was  used  to 
redistribute  the  explained  variance  among  the  two  factors. 

The  factor  loadings  resulting  from  the  varimax  rotation  are 
shown  in  Table  27. 

Chapter  Summary 

This  chapter  provided  a  summary  presentation  of  the 
results  of  this  research  effort.  The  results,  along  with 
the  interview  data,  form  the  basis  for  the  responses  to  the 
three  research  questions  posed  in  Chapter  I.  In  the 
following  chapter,  each  research  question  is  restated  and 
answered. 
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Table  27. 


Factor  Loadings 


Variable  Name 

Factor  1 

Factor  2 

Communicability 

(Did  not  correlate  significantly 
with  "Willingness  to  Adopt") 

Appropriateness 
of  Outputs 

.4525 

.5526 

Appropriateness 
of  Mathematics 

.0688 

.6191 

Availability  of 
Inputs 

-.0025 

.8609 

Upward 

Compat ib i 1 i ty 

.7393 

.0253 

Downward 

Compatibility 

.7688 

.4034 

Process 

Compatibility 

.5192 

.6640 

Uncertainty 

Absorption 

.9264 

.1860 

Information 

Value 

.8487 

.1769 

Process  Value 

.5300 

.6532 
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V.  Analysis  and  Discussion 


Overview 

This  chapter  provides  an  analysis  and  discussion  of  the 
results  presented  in  Chapter  IV.  Both  the  survey  data  and 
the  information  gathered  in  interviews  was  used  to  develop 
these  results.  In  this  chapter,  each  research  question  is 
listed,  followed  by  a  detailed  response.  The  research 
findings  presented  in  this  chapter  are  applicable  to  all  four 
AFWAL  laboratories  since  the  Kruskal-Wallis  analysis  of 
variance  indicated  that  there  were  no  statistically  signif¬ 
icant  differences  between  the  responses  between  the  four 
laboratories. 

Research  Question  1 

What  methods  do  managers  use  to  select  R&D  projects  and 
allocate  laboratory  resources  at  the  Air  Force  Wright 
Aeronautical  Laboratories? 

Use  ot  P&P  projqqt  Select jap  Techniques.  Formal 
project  selection  techniques  play  a  minor  role  in  the  project 
selection  process  at  AFWAL.  Furthermore,  the  respondents,  in 
general,  are  not  familiar  with  most  R&D  project  selection 
techniques.  Over  half  of  the  respondents  indicated  that  they 
were  not  familiar  with  10  out  of  the  15  techniques  listed  in 
the  survey.  The  techniques  which  were  recognized  by  at 
least  half  of  the  respondents  are  listed  in  Table  28. 
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Table  28. 


Ranking  of  R&D  Project  Selection 
Techniques  By  Familiarity  * 


Technique 

Familiar  With 

Frequency 

and/or  Use 

Percent 

Delphi 

26 

81.2 

Cost/Benefit  Ratios 

24 

75.0 

Decision  Analysis 

24 

75.0 

Risk  Analysis/Monte 

Carlo  Simulation 

18 

56.3 

Payback  Period 

17 

53.2 

*  Includes  only  those  techniques  which  were  recognized  by 
50  percent  or  more  of  the  respondents 


As  expected,  the  use  of  R&D  project  selection  techniques 
was  also  limited.  Only  25%  of  the  respondents  indicated  that 
they  used  a  formal  R&D  project  selection  technique. 

Furthermore,  only  seven  of  the  techniques  listed  in  the 
survey  were  used  by  any  of  the  respondents.  The  seven 
techniques  are  shown  in  Table  29.  They  are  rank-ordered 
based  on  their  rate  of  use. 

Because  the  education  level  of  the  respondents  was 
high,  the  low  level  of  familiarity  with  these  techniques  was 
surprising.  Table  10  showed  that  75  percent  of  the  respon¬ 
dents  had  obtained  at  least  a  Master's  Degree,  and  a  very 
large  percentage  of  respondents  had  completed  course-work  in 
related  topics  such  as  MS/OR  (81.3  percent)  and  Laboratory 
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Table  29. 

Ranking  of  R&D  Project  Selection  Techniques  By  Use 


Technique 

Used  Regularly 

Frequency  Percent 

Checklists 

5 

15.6 

Delphi 

5 

15.6 

Cost/Benefit  Ratios 

3 

9.4 

Payback  Period 

2 

6.3 

Net  Present  Value/ 

Internal  Rate  of  Return 

1 

3.1 

Goal  Programming 

1 

3.1 

Decision  Analysis 

1 

3.1 

Management  (84.4  percent).  Although  the  familiarity  and 


usage  were  low,  the  respondents  did  indicate  a  desire  to 
learn  more  about  these  techniques.  Over  80  percent  of  the 
respondents  indicated  that  they  would  be  willing  to  attend  a 
course  or  seminar  on  the  topic  of  R&D  project  selection 
techniques. 

It  should  also  be  noted  that  during  the  interview 
process  there  were  several  indications  that  managers  utilized 
processes  which  were  very  similar  to  some  formal  project 
selection  techniques.  However,  most  managers  did  not 
indicate  knowledge  of  the  names  of  the  formal  techniques. 

This  was  especially  true  of  those  techniques  which  utilize 
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qualitative  selection  criteria,  such  as  checklists,  scoring 
models,  and  Delphi  techniques. 

Pressure  to  Quantify  R&D  Expenditures.  Table  24 
indicated  that  AFWAL  managers  experienced  moderate  amounts  of 
pressure  from  outside  sources  (e.g.  Systems  Command,  Air 
Force  Office  of  Scientific  Research)  to  quantify  their  R&D 
expenditures.  On  a  seven-point  Likert  scale,  over  half  of 
the  respondents  marked  either  1,  2,  or  3,  with  the  value  1 
corresponding  to  "Extreme  Pressure,"  and  the  value  7  to  "No 
Pressure."  Furthermore,  only  25.1  percent  of  the  respondents 
marked  5,  6,  or  7  (28.1  percent  marked  the  middle  value 
"Some  Pressure") . 

On  the  other  hand,  the  amount  of  pressure  being  exerted 
from  within  AFWAL  to  quantify  R&D  expenditures  was  somewhat 
less,  as  shown  by  Table  23.  In  this  case,  only  31.3  percent 
of  the  respondents  marked  either  1,  2,  or  3,  while  40.7 
percent  marked  5,  6,  or  7  (28.1  percent  marked  the  middle 
value  "Some  Pressure") . 

Therefore,  while  it  is  clear  that  managers  are  faced 
with  moderate  amounts  of  pressure  to  quantify  their  R&D 
expenditures,  especially  from  sources  outside  AFWAL,  they  do 
not  choose  to  utilize  formal  R&D  project  selection  techniques 
to  respond  to  this  pressure.  This  finding  prompts  two 
questions.  First,  how  are  R&D  projects  evaluated  if  formal 
techniques  are  not  used?  Second,  why  would  a  manager  decide 
against  using  these  techniques?  The  first  question  is 
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addressed  below.  The  second  question  will  be  addressed  in 
the  response  to  Research  Question  3. 

The  R&D  Project  Selection  Process.  Although  formal 
project  selection  techniques  are  not  widely  used  at  AFWAL, 

R&D  managers  in  all  four  laboratories  indicated  that  they  do 
rely  on  a  systematic  process  to  evaluate  project  proposals 
and  allocate  laboratory  resources.  This  section  provides  a 
description  of  the  R&D  project  selection  process  at  AFWAL. 

It  is  followed  by  a  discussion  of  several  specific  charac¬ 
teristics  of  the  process.  Because  intuition  and  expert 
judgment  often  play  a  significant  role  in  the  R&D  project 
selection  process,  the  interviews  conducted  in  support  of 
this  research  were  especially  helpful  in  obtaining  a  complete 
description  of  the  process  (48:28). 

Prior  to  this  discussion,  background  information  will  be 
presented  concerning  the  categories  of  R&D  conducted  at 
AFWAL,  and  the  organizational  structure  of  a  typical  labor¬ 
atory. 

There  are  four  major  categories  of  research  conducted 
at  AFWAL:  basic  research;  exploratory  development;  advanced 
development;  and  production  base  support  (32:2).  These 
categories  of  research  are  commonly  referred  to  by  the 
program  element  number  under  which  they  are  funded.  Overall, 
the  primary  emphasis  is  on  exploratory  and  advanced  devel¬ 
opment  work.  However,  the  Materials  Laboratory  performs  a 
significant  amount  of  research  in  the  area  of  manufacturing 
technology  to  support  the  production  base. 
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Table  30  shows  the  funding  appropriations  for  each  major 
program  element  as  a  percentage  of  each  laboratory's  total 
expenditures.  The  data  in  Table  30  includes  direct  labora¬ 
tory  funds,  as  well  as  funds  provided  by  the  Air  Force 
Office  of  Scientific  Research  (AFOSR)  for  basic  research. 
Research  funded  by  other  external  organizations  (including 
other  Air  Force  and  non-Air  Force  organizations)  is  not 
included.  The  engineering  development  (6.4)  work  conducted 
by  the  Avionics  and  Flight  Dynamics  Laboratories  is  in 
support  of  research  efforts  which  are  in  transition  from  the 

Table  30. 


Proportionate  Laboratory  Funding  by  Program  Element 


AFWAL  Laboratory 
(Percent  of  Total  Funds) 

Program  Element 

Avionics 

Flight 

Dynamics 

Materials 

Aero- 

Prop. 

Basic 

Research  (6.1) 

2.7 

4.4 

3.9 

4.6 

Exploratory 
Development  (6.2) 

25.4 

43.8 

31.7 

42.4 

Advanced 

Development  (6.3) 

50.6 

43.3 

10.8 

50.4 

Engineering 
Development  (6.4) 

17.8 

5.2 

0.0 

0.0 

Production  Base 
Support  (7.8) 

0.0 

0.0 

48.9 

0.0 

Other 
(6.5,  7.1) 

3.5 

3.3 

4.7 

2.6 

Total 

100.0 

100.0 

100.0 

100.0 

Source  (11) 
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laboratory  to  a  program  office.  Miscellaneous  expenditures 
include  such  items  as  small  business  innovative  research 
efforts,  and  the  acquisition  and  maintenance  of  embedded 
computer  resources.  Beginning  with  basic  research,  each 
category  of  R&D  becomes  progressively  more  application- 
oriented.  The  purpose  of  this  data  is  to  demonstrate  the 
emphasis  placed  on  each  category  of  research  within  the 
laboratories.  With  its  heavy  emphasis  on  exploratory 
development,  advanced  development,  and  production  base 
support,  it  is  clear  that  the  R&D  conducted  at  AFWAL  is  very 
"mission  oriented."  As  noted  in  Chapter  II,  the  mission- 
oriented  nature  of  public  sector  R&D  organizations  has  had  an 
effect  on  the  implementation  of  R&D  project  selection 
techniques.  This  point  will  be  discussed  in  more  detail 
later  in  this  chapter. 

The  organizational  structure  of  a  typical  AFWAL 
laboratory  is  depicted  in  Figure  2.  The  four  laboratories 
range  in  size  from  350  employees  to  over  700.  This  back¬ 
ground  on  the  funding  appropriations  and  organizational 
structure  of  AFWAL  will  help  to  clarify  the  following 
discussion  of  the  R&D  project  selection  process. 

The  project  selection  process  at  each  laboratory  is 
very  similar,  and  generally  occurs  annually.  The  process  is 
timed  to  support  the  submission  of  the  Program  Objective 
Memorandum. 

Technical  teams  consisting  of  engineers  and  scientists 
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Figure  2.  Organizational  Structure  of  a  Typical  AFWAL 
Laboratory  (Adapted  from  27) 


are  formed  to  provide  progress  reports  and  proposals  for  new 
projects  to  upper  management.  The  teams  are  formed  around  a 
specific  technological  area.  The  members  of  the  technical 
teams  mayor  may  not  be  from  the  same  branch,  depending  on  the 
amount  of  technical  integration  required.  The  technical  area 
teams  are  responsible  for  all  categories  of  R&D  within  their 
discipline  (i.e.  basic  research,  exploratory  development, 
advanced  development,  and  production  base  support,  if 


68 


applicable) .  The  information  developed  by  the  technical 
teams  is  provided,  in  briefing  form,  to  the  ’’corporate 
board . " 

The  corporate  board  is  a  high-level  management  group 
typically  comprised  of  the  Director,  Deputy  Director,  Chief 
Scientist,  Director  of  Planning,  and  Division  Chiefs  within 
the  laboratory.  After  reviewing  briefings  from  each  of  the 
technical  teams,  the  corporate  board  then  makes  project 
selection  decisions  for  the  laboratory. 

Although  the  project  selection  process  was  found  to  be 
similar  for  each  laboratory,  the  process  used  by  the 
Materials  Laboratory  was  found  to  be  somewhat  more  struc¬ 
tured.  In  the  Materials  Laboratory  process,  the  corporate 
board  provides  each  technical  team  with  feedback  in  the  form 
of  written  questions  following  their  presentation.  The 
technical  teams  are  then  required  to  respond  to  these 
questions  in  a  follow-up  briefing  before  final  decisions  are 
made . 

Despite  minor  differences  between  the  project  selection 
process  in  each  laboratory,  the  corporate  board  by  and  large 
relies  on  intuition  and  expert  judgment  as  the  primary  means 
of  making  project  selection  decisions. 

The  preceding  discussion  provided  a  brief  overview  of 
the  project  selection  process.  In  the  following  sections, 
several  of  the  more  pertinent  aspects  of  the  project 
selection  process  will  be  discussed  in  detail. 
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Use  of  Off-sites.  Several  respondents  indicated  that 
the  corporate  board  often  held  project  selection  meetings  in 
"off-site"  locations.  These  "off-sites"  are  typically  held 
at  local  convention  centers ,  and  are  designed  to  promote 
frank  and  honest  discussion  during  the  project  selection 
process . 

Technology  Push  and  Requirements  Pull.  From  interviews 
with  R&D  managers,  it  was  readily  apparent  that  research  at 
AFWAL  is  heavily  influenced  by  "requirements  pull"  as 
opposed  to  "technology  push."  Thus,  it  is  important  for 
scientists  and  engineers  to  link  their  R&D  projects  to 
existing  or  planned  Air  Force  programs  to  increase  the 
chances  that  their  projects  will  be  continued  or  initiated  by 
the  corporate  board. 

As  shown  in  Table  30,  it  was  found  that  the  majority  of 
funding  is  concentrated  in  exploratory  and  advanced  develop¬ 
ment,  and  production  base  support,  rather  than  basic 
research.  In  fact,  most  managers  indicated  that  demonstrat¬ 
ing  that  a  R&D  project  benefits  a  specific  Air  Force  program 
is  a  primary  selection  criterion.  As  such,  it  is  probably 
not  coincidental  that  three  of  the  seven  most  frequently  used 
project  selection  techniques  are  designed  to  evaluate 
projects  in  terms  of  economic  benefits  (see  Table  29) . 

Impact  of  Budget  Constraints.  There  was  nearly 
unanimous  agreement  among  those  interviewed  that  budget 
constraints  have  a  significant  impact  on  the  project 
selection  process.  For  the  most  part,  respondents  indicated 
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that  budget  constraints  force  managers  to  select  those  R&D 
projects  which  are  less  risky,  and  provide  more  immediate 
short-term  benefits.  Because  of  this,  some  managers  felt 
that  budget  constraints  make  the  job  of  project  selection 
easier.  However,  most  managers  felt  that  budget  constraints 
challenge  them  to  achieve  a  balanced  portfolio  of  R&D 
projects,  which  included  both  long-term,  high-risk  projects, 
as  well  as  more  immediate,  high-payoff  projects. 

In  summary,  it  was  found  that  formal  project  selection 
techniques  are  not  widely  used  by  AFWAL  managers,  despite 
some  pressure  to  quantitatively  justify  R&D  expenditures. 

It  was  also  found  that,  although  R&D  project  selection 
techniques  are  not  widely  used,  all  four  laboratories 
employed  a  systematic  project  selection  process.  Further¬ 
more,  it  was  shown  that  expert  judgment  plays  a  primary  role 
in  the  selection  process,  and  that  managers  often  utilize 
"off-site"  meetings  to  encourage  candid  discussion.  Finally, 
it  was  shown  that  both  "requirements  pull"  and  budget 
constraints  have  a  significant  influence  on  the  project 
selection  process. 

Research  Question  2 

What  are  the  attitudes  and  perceptions  of  AFWAL  R&D 
managers  toward  the  use  of  formal  R&D  project  selection 
techniques? 

The  reply  to  this  research  question  is  derived  from  the 
responses  to  survey  questions  17  through  25,  as  well  as  from 
the  interviews.  Survey  questions  17  through  25  measured  the 
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attitudes  and  perceptions  of  the  respondents  concerning  their 
project  selection  processes  and  the  potential  benefit  of 
formal  R&D  project  selection  techniques.  The  interviews  were 
especially  helpful  in  determining  why  managers  are  reluctant 
to  utilize  these  techniques.  The  results  are  discussed  below. 

Adequacy  of  Existing  System.  A  majority  of  respondents 
(62.5  percent)  indicated  that  they  were  generally  satisfied 
with  the  current  project  selection  process  in  their  labor¬ 
atory  (survey  question  29) .  However,  approximately  two- 
thirds  of  the  respondents  felt  that  the  process  needs  at 
least  some  improvement  (Table  16) .  From  this,  it  can  be 
inferred  that  AFWAL  managers  are  somewhat  motivated  to 
investigate  the  capabilities  of  R&D  project  selection 
techniques . 

Vti-Uty  Of  R&D  .Project  gejection  Tecim.Lqueg.  Tables  16 
and  17  showed  that  a  majority  of  managers  felt  that  a  project 
selection  technique  would  be  beneficial  to  them  (68.8 
percent);  68.8  percent  also  felt  that  such  a  technique  would 
be  beneficial  to  their  subordinates.  However,  an  actual 
decision  to  utilize  such  techniques  brought  mixed  responses. 
The  data  in  Table  19  showed  that  managers  felt  somewhat 
strongly  about  an  organizational  decision  to  implement  R&D 
project  selection  techniques  for  all  project  selections  - 
46.9  percent  of  the  respondents  felt  that  such  a  decision  was 
relatively  important,  while  34.5  perceived  it  to  be  rela¬ 
tively  unimportant  (18.8  percent  were  uncertain) .  The 
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respondent's  personal  recommendations  on  such  a  decision  were 
equally  divided,  with  37.6  percent  for,  37.6  percent  against, 
and  the  remainder  undecided. 

The  response  to  the  question  concerning  the  use  of 
project  selection  techniques  on  half  of  the  respondent's 
project  selections  showed  similar  results.  Table  21  shows 
that  47  percent  of  the  respondents  felt  that  an  organiza¬ 
tional  decision  to  implement  these  techniques  was  important, 
while  34.4  percent  thought  it  to  be  relatively  unimportant 
(18.8  percent  undecided).  Furthermore,  43.8  percent  of  the 
respondents  were  generally  in  favor  of  using  formal  tech¬ 
niques  on  half  of  their  projects,  while  28.1  percent  were  not 
(25  percent  undecided) . 

In  addition,  respondents  were  skeptical  about  the 
probability  of  developing  a  project  selection  technique  that 
would  greatly  enhance  the  effectiveness  of  their  project 
selection  decisions  (35.5  percent  found  it  likely;  46.9 
percent  found  it  unlikely;  16.1  percent  were  uncertain). 

In  summary,  respondents  were  generally  satisfied  with 
their  current  project  selection  process;  however,  many  felt 
it  needed  some  improvement.  Furthermore,  although  a 
majority  of  the  respondents  indicated  that  a  formal  project 
selection  technique  would  be  beneficial,  most  felt  that  it  is 
unlikely  that  such  a  technique  could  be  developed  that  would 
adequately  consider  the  wide  range  of  pertinent  selection 
criteria.  Finally,  it  can  be  inferred  that  managers  are,  for 
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the  most  part,  skeptical  about  the  promise  of  formal  project 
selection  techniques. 

The  final  research  question  sought  to  identify  the 
specific  variables  and  factors  that  affect  the  adoption  of 
R&D  project  selection  techniques  by  AFWAL  managers. 

Research  Question  3 

What  factors  affect  management's  willingness  to  adopt 

formal  R&D  project  selection  techniques? 

Of  the  ten  variables  tested,  nine  were  found  to 
correlate  with  "willingness  to  adopt"  at  a  .05  level  of 
significance  (see  Table  26) .  The  variables  used  in  this 
research  were  previously  defined  in  Table  7.  The  variable 
"communicability"  did  not  correlate  significantly  with 
"willingness  to  adopt." 

Two  factors  were  extracted  based  on  their  eigenvalues 
(greater  than  or  equal  to  1.0)  in  the  initial  principal 
components  extraction.  These  two  factors  account  for  66.7 
percent  of  the  total  variance  of  the  data.  A  varimax 
rotation  was  used  to  distribute  the  variance  among  the  two 
factors.  It  was  determined  apriori  that  variables  which  had 
a  factor  loading  of  .5  or  greater  would  be  included  in  that 
factor.  The  nine  variables  loaded  as  shown  in  Table  31.  In 
keeping  with  the  decision  rule  for  assigning  variables  to 
factors,  the  variables  "process  value"  and  "process  compat¬ 
ibility"  were  included  in  both  factors. 

As  noted  by  Kachigan,  factor  names  should  "represent  a 
common  element  or  abstraction  of  the  individual  variables 
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which  loaded  highly  on  the  factor"  (18:393).  After  analyzing 
each  factor,  names  were  selected  as  shown  in  Table  31. 

The  variables  which  comprise  the  first  factor  are 
related  to  the  process  of  organizational  decision  making. 

Table  31. 

Factor  Names  and  Variable  Loadings 


Factors  and  Variables 

Factor  Loadings 

Factor  1:  Organizational 
Decision  Making 

Uncertainty  Absorption 

.9296 

Information  Value 

.8487 

Downward  Compatibility 

.7688 

Upward  Compatibility 

.7393 

Process  Value  * 

.5300 

Process  Compatibility  * 

.5192 

Factor  2:  R&D  Project 
Selection  Techniques 

Availability  of 

Inputs 

.8609 

Process  Compatibility  * 

.6640 

Process  Value  * 

.6534 

Appropriateness 
of  Mathematics 

.6191 

Appropriateness  of 

Outputs 

.5526 

*  Variable  loaded  on  both  factors 
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This  first  factor  shows  that,  as  a  group,  the  respon¬ 
dents  do  not  feel  that  formal  project  selection  techniques 
will  reduce  uncertainty  about  decisions  provide  additional 
information  or  greatly  facilitate  communication  within  the 
organization.  Furthermore,  this  factor  shows  that  the 
respondents  do  not  feel  that  a  formal  project  selection 
technique  is  necessary,  nor  would  it  be  compatible  with  the 
existing  process. 

The  variables  which  comprise  the  second  factor  are 
primarily  related  to  the  R&D  project  selection  techniques 
themselves.  This  factor  shows  that  the  respondents  feel 
that  1)  the  input  data  required  by  formal  techniques  would  be 
difficult  to  develop;  2)  that  the  mathematics  involved  with 
the  quantification  of  project  selection  decisions  is  more 
complex  than  necessary;  and  3)  that  the  outputs  from  these 
techniques  would  not  yield  information  which  is  relevant  to 
the  project  selection  process. 

Although  these  results  show  that  the  respondents 
generally  have  negative  attitudes  toward  capabilities  of  R&O 
project  selection  techniques,  it  must  be  noted  that  these 
results  represent  an  initial  set  of  variables  which  are  based 
only  on  the  respondents'  perceptions.  Research  conducted  by 
Souder  showed  that  a  "negative-to-positive  shift  in  adoption 
attitudes"  occurred  when  respondents  were  provided  with 
education  and  training  in  the  use  of  these  techniques 
(45:75).  Thus,  until  this  has  been  accomplished,  the 
stability  of  the  adoption-affecting  variables  can  not  be 
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verified.  Nonetheless,  these  results  serve  as  a  basis  for 
further  analysis  by  providing  a  characterization  of  the 
existing  project  selection  process  and  establishing  an 
initial  set  of  adoption-affecting  variables. 

Chapter  Summary 

This  chapter  provided  an  analysis  and  discussion  of  the 
findings  of  the  research  effort.  Using  the  results  of  the 
survey  data  presented  in  Chapter  IV,  as  well  as  the  data 
collected  in  the  interviews,  each  of  the  three  research 
questions  was  addressed  in  detail. 

The  following  chapter  provides  a  summary  of  the 


conclusions  that  can  be  drawn  from  this  research,  as  well  as 
suggested  recommendations  for  future  research. 


VI.  Conclusions  and  Recommendations 

This  chapter  provides  a  summary  of  the  key  findings  of 
this  research  effort.  Recommendations  pertaining  to  both  the 
R&D  project  selection  process  at  AFWAL  and  this  research 
effort  are  also  suggested. 

Conclusions 

1.  From  the  survey  data,  it  can  be  concluded  that 
formal  R&D  project  selection  techniques  play  a  minor  role  in 
the  project  selection  process  at  AFWAL.  Only  25  percent  of 
the  respondents  indicated  that  they  used  a  formal  technique 
as  part  of  their  project  selection  process.  Furthermore,  as 
a  group,  managers  at  AFWAL  are  not  familiar  with  many  of  the 
most  popular  techniques.  Of  the  15  techniques  listed  in  the 
survey,  only  five  were  recognized  by  more  than  half  of  the 
respondents.  The  levels  of  familiarity  with,  and  usage  of 
R&D  project  selection  techniques  did  not  differ  significantly 
between  the  four  laboratories. 

2.  Although  AFWAL  managers  do  not  rely  heavily  on 
formal  project  selection  techniques,  they  do  utilize 
systematic  methods  for  reviewing  technical  progress, 
selecting  R&D  projects,  and  allocating  laboratory  resources. 

The  project  selection  processes  utilized  by  the 
laboratories  were  found  to  be  similar.  At  the  laboratory 
level,  technical  progress  and  proposals  for  new  R&D  projects 
are  formally  reviewed  on  an  annual  basis.  Technical  teams, 
comprised  of  laboratory  scientists  and  engineers,  are 
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responsible  for  developing  this  information,  and  providing 
it,  in  briefing  format,  to  a  "corporate  board."  The 
corporate  board  typically  consists  of  the  Director,  Deputy 
Director,  Chief  Scientist,  and  Division  Chiefs  of  each 
laboratory.  The  corporate  board  generally  relies  on  expert 
judgment  to  reach  a  consensus  on  project  selection  decisions. 
Several  managers  noted  that  "off-site"  meetings  were  used  to 
facilitate  strategic  planning. 

3 .  Managers  felt  that  there  was  a  moderate  amount  of 
pressure  being  exerted  from  outside  sources  (e.g.,  Air  Force 
Office  of  Scientific  Research,  Air  Force  Systems  Command)  to 
quantitatively  justify  their  R&D  expenditures.  The  amount  of 
pressure  being  generated  from  within  AFWAL  was  perceived  to 
be  somewhat  less. 

4.  The  R&D  project  selection  process  at  AFWAL  is 
heavily  influenced  by  "requirements  pull,"  i.e.,  the  mission 
requirements  of  the  Air  Force  drive  the  project  selection 
process.  For  advanced  development  (6.3)  work  in  particular, 
the  objective  of  management  becomes  one  of  evaluating  the 
effectiveness  of  R&D  projects  undertaken  in  response  to  a 
specific  Air  Force  need,  as  opposed  to  selecting  R&D  projects 
for  their  potential  to  advance  scientific  knowledge. 

This  emphasis  on  exploratory  and  advanced  development, 
and  production  base  support  (the  percentage  of  laboratory 
expenditures  devoted  to  these  areas  is  shown  in  Table  30) , 
indicates  that  AFWAL  R&D  is  driven  primarily  by  "requirements 
pull"  as  opposed  to  "technology  push."  As  noted  in  Chapter 
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II,  the  mission-oriented  nature  of  public  sector  R&D  often 
places  unique  constraints  on  the  R&D  manager,  especially  in 
terms  of  quantifying  research  benefits.  This  fact  may  place 
constraints  on  the  applicability  of  many  R&D  project 
selection  techniques.  Thus,  due  to  the  nature  of  the 
military  R&D  environment,  R&D  expenditures  are  often 
justified  by  the  establishment  of  formal  links  to  the  "user" 
such  as  memorandums  of  agreement. 

5.  Budget  constraints  have  a  significant  impact  on  the 
project  selection  process  at  AFWAL.  In  general,  managers 
felt  that  budget  constraints  force  them  to  focus  on  short¬ 
term,  high-payoff  projects  which  exhibit  low  technical  risk. 
When  faced  with  budget  constraints,  managers  noted  that  the 
challenge  is  to  maintain  a  balanced  research  portfolio, 
consisting  of  both  long-  and  short-term  research  efforts,  to 
ensure  that  the  laboratory  maintains  an  innovative  edge,  and 
does  not  become  myopic. 

6.  Respondents  indicated  that  they  were,  in  general, 
satisfied  with  the  current  R&D  project  selection  process  in 
their  laboratories.  Although  many  felt  that  a  technique 
designed  to  help  them  evaluate  and  select  R&D  projects  would 
be  very  beneficial,  a  majority  felt  that  the  likelihood  of 
developing  such  a  technique  was  low.  Even  if  such  tech¬ 
niques  were  developed,  respondents  were  divided  as  to  whether 
they  would  implement  them  in  their  project  selection  process. 
The  attitudes  and  perceptions  of  the  respondents  did  not 
differ  significantly  between  the  four  laboratories. 
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7.  Nine  variables  were  found  to  be  associated  with 
management's  willingness  to  adopt  formal  R&D  project 
selection  techniques  at  a  .05  level  of  significance. 

The  variables  are  1)  the  appropriateness  of  the  outputs 
provided  by  project  selection  techniques;  2)  the  appropriate¬ 
ness  of  the  mathematics  used  to  derive  the  outputs;  3)  the 
availability  of  input  data;  4)  the  ability  of  project 
selection  techniques  to  facilitate  upward  communication;  5) 
the  ability  of  project  selection  techniques  to  facilitate 
downward  communication;  6)  the  compatibility  of  formal 
techniques  with  the  existing  project  selection  process;  7) 
the  informational  value  of  project  selection  techniques;  8) 
the  ability  of  project  selection  techniques  to  remove 
uncertainty  in  the  decision-making  process;  and  9)  the 
respondents'  need  for  structure  in  their  own  decision 
process . 

These  nine  variables  constitute  an  initial  set  of 
adoption-affecting  variables,  based  on  the  respondents 
perceptions  about  formal  R&D  project  selection  techniques. 
They  were  factored  into  two  groups,  which  accounted  for  66.7 
percent  of  the  variation  of  the  data.  The  results  showed 
that  the  factors  that  affect  management's  willingness  to 
adopt  formal  project  selection  techniques  are  related  to:  1) 
the  effect  of  formal  project  selection  techniques  on  the 
organizational  decision-making  process;  and  2)  the  character¬ 
istics  of  the  R&D  project  selection  techniques  themselves. 
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The  results  also  indicate  that  the  respondents,  based 
on  their  initial  perceptions  and  current  level  of  knowledge, 
generally  feel  that  formal  R&D  project  selection  techniques 
have  limited  utility  as  a  management  tool. 

At  this  point,  it  is  important  to  reiterate  the  results 
of  research  conducted  by  Dr.  William  Souder  on  this  subject. 
Dr.  Souder  found  that  negative-to-positive  attitude  shifts 
occurred  after  respondents  attended  training  sessions  on  the 
theory  and  application  of  formal  project  selection  techniques 
(45:75).  A  recommendation  concerning  training  is  discussed 
in  the  next  section. 

This  investigation  of  the  R&D  project  selection  process 
at  AFWAL  provides  1)  a  characterization  of  the  project 
selection  process;  2)  a  characterization  of  the  attitudes  and 
perceptions  of  AFWAL  managers  toward  the  utility  of  R&D 
project  selection  techniques;  and  3)  the  identification  of 
an  initial  set  of  variables  which  affect  management's 
willingness  to  adopt  R&D  project  selection  techniques. 
Together,  this  information  serves  as  a  basis  for  follow-on 
research.  When  the  stability  of  the  adoption-affecting 
variables  set  is  established,  the  resulting  data  will  provide 
the  information  necessary  to  evaluate  and/or  develop  R&D 
project  selection  techniques  that  will  respond  to  the  needs 
of  AFWAL  managers,  achieve  successful  implementation  in  the 
organization,  and  enhance  the  effectiveness  of  R&D  project 
evaluation  and  selection. 
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General  Recommendations 


1.  Given  the  respondents*  low  level  of  familiarity 
with  formal  project  selection  techniques  and  Dr.  Souder's 
research  results  concerning  negative-to-positive  attitude 
shifts,  the  first  recommendation  is  that  efforts  be  made  to 
increase  AFWAL  management's  understanding  of  the  uses  of 
formal  R&D  project  selection  techniques,  with  an  emphasis  on 
techniques  designed  to  incorporate  qualitative  input  data, 
such  as  checklists,  scoring  models,  and  Delphi  methods. 
Management  should  be  advised  that  a  new  generation  of 
techniques  has  evolved  which  may  be  more  responsive  to  their 
needs  (46:36). 

Informing  managers  about  the  potential  of  these  tech¬ 
niques  could  be  accomplished  through  seminars,  through  the 
addition  of  courses  in  the  Professional  Continuation  Program 
on  base,  or  through  the  individual  efforts  of  AFWAL  managers. 
The  literature  review  contained  in  this  document  provides  a 
brief  overview  of  four  major  categories  of  R&D  project 
selection  techniques.  In  addition,  many  of  the  journals  and 
articles  listed  in  the  bibliography  offer  excellent  reviews 
of  the  latest  R&D  project  selection  techniques. 

2.  The  semi-structured  nature  of  the  R&D  project 
selection  process  may  lend  itself  to  applications  of 
computer-based  decision  support  systems  technology.  In 
reference  to  computer-based  decision  support  systems,  Gordon 
Davis  and  Margrethe  Olson  note  the  following: 
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The  main  payoff  of  computer  support  is  for  semi- 
structured  problems,  where  parts  of  the  analysis  can  be 
systematized  for  the  computer,  but  where  the  decision 
maker's  insight  and  judgment  are  needed  to  control  the 
process.  (5:368) 

As  such,  a  second  recommendation  is  that  AFWAL  manage¬ 
ment  investigate  the  applicability  of  computer-based  decision 
support  systems  as  a  means  of  enhancing  the  R&D  project 
selection  and  evaluation  process. 

Recommendations  for  Further  Study 

1.  Although  project  selection  decisions  are  made  by  a 
"corporate  board"  at  each  laboratory,  the  responsibility  for 
deriving  the  information  used  in  these  decisions  is  that  of 
the  scientists  and  engineers  comprising  the  "technical 
teams."  It  is  recommended  that  these  groups  be  included  in 
any  follow-on  research  efforts.  By  including  a  larger  number 
of  respondents,  it  would  also  allow  the  investigator  to  test 
a  larger  set  of  adoption-affecting  variables  without  sacri¬ 
ficing  sampling  adequacy.  However,  when  expanding  the  sample 
group,  care  must  be  exercised  so  that  only  those  who 
influence  the  project  selection  process  are  chosen. 

2.  As  noted  above,  the  semi-structured  nature  of  the 
project  selection  process  suggests  potential  applications  for 
computer-based  decision  support  systems.  Research  into  the 
informational  requirements  of  R&D  managers  as  they  pertain  to 
the  project  selection  process  could  provide  further  informa¬ 
tion  on  this  subject. 
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3 .  To  increase  organizational  awareness  about  the 
potential  of  R&D  project  selection  techniques,  seminars 
should  be  conducted  to  inform  managers  of  these  techniques. 
Following  the  seminar,  the  survey  instrument  provided  in  this 
research  effort  should  be  re-administered  to  evaluate  the 
stability  of  the  adoption-affecting  variables  set.  The 
continuation  of  this  research  effort  would  be  useful  in 
helping  to  identify  a  R&D  project  selection  technique  that 
would  likely  be  adopted  and  used  by  a  laboratory. 

4.  As  noted  in  the  fourth  conclusion  above,  as  R&D 
projects  progress  from  basic  research  to  advanced  devel¬ 
opment,  the  function  of  management  becomes  one  of  evaluating 
the  effectiveness  of  an  R&D  project  that  addresses  an  Air 
Force  need,  rather  than  selecting  a  portfolio  of  projects. 
Therefore,  an  investigation  of  R&D  project  evaluation  tech¬ 
niques  which  are  applicable  to  mission-oriented  agencies  may 
encourage  the  Air  Force  manager  to  adopt  a  formal  R&D 
project  selection  technique. 
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Appendix  A:  R&D  Project  Selection  Survey 


■«RLY  TO 
ATT*  OA: 


DEPARTMENT  OF  THE  AIR  FORCE 
air  *crce  wright  aeronautical  laboratories  iaesci 

WRiGhT-RATTERSON  AIR  PORCE  BASE.  OHIO  4SA33-BS23 


1  6  MAY  JSS8 


subjbct:  R&D  Project  Selection  Survey 


to:  Diacribucion 


1.  Th«  attached  survey  will  be  used  to  investigate  the  R&O 
project  selection  and  resource  allocation  process  here  at  the 
Air  Force  Wright  Aeronautical  Laboratories .  This  information 
is  being  collected  by  a  graduate  student  at  the  Air  Force 
Institute  of  Technology.  I  have  been  briefed  as  to  the 
nature  of  this  research,  and  request  that  you  take  a  few 
minutes  to  participate  in  this  survey.  I  have  been  assured 
that  all  responses  will  be  strictly  confidential. 

2.  The  nature  of  this  research  requires  that  our  top  level 
managers  respond  to  these  questions.  It  is  requested  that  you 
da  not  delegate  this  survey  to  your  staff. 

3.  In  addition  to  responding  to  the  survey,  some  of  you  may 
be  asked  to  participate  in  a  short  interview.  I  also  ask 
that  you  comply  with  this  request  if  at  all  possible.  Any 
questions  concerning  the  attached  survey  should  be  addressed 
to  the  researcher,  Captain  Jon  Congdon,  at  (313)  236-4645. 


R&D  Project  Selection  Survey 


UNHID  STATU  AM  PORC* 


SEPTEMBER  18,1947 
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R&D  Proi ect  Selection  Survey 


Purpose:  The  purpose  of  this  survey  is  to  investigate  the 

R&D  project  selection  and  resource  allocation  process  at  the 
Air  Force  Wright  Aeronautical  Laboratories.  Several  formal 
techniques  have  been  developed  to  assist  R&D  managers  in 
making  project  selection  decisions.  This  survey  is  designed 
to  measure  your  opinions  concerning  the  utility  of  these 
techniques . 

A  formal  project  selection  technique  is  defined  as  any  model 
or  algorithm  used  to  systematically  assign  values  to 
individual  R&D  projects  or  groups  of  projects.  Formal 
project  selection  techniques  can  be  grouped  into  the 
following  categories:  1)  subjective  methods;  2)  economic 
techniques;  3)  risk  analysis  techniques;  and  4)  math 
programming  models.  A  list  of  project  selection  techniques 
appears  in  questions  1  through  15  of  this  survey. 

Confidentiality;  Your  responses  to  this  survey  will  be 
strictly  confidential. 

Survey  Responses :  None  of  the  survey  questions  are  intended 

for  you  to  look  up  any  information.  Only  your  opinions  are 
sought.  After  carefully  reading  the  questions,  it  is 
suggested  that  you  respond  to  them  quickly.  If  you  would 
like  to  make  any  comments,  please  feel  free  to  mark  them 
directly  on  the  survey. 

After  completing  this  survey,  please  place  it  in  the  return 
envelope  provided  and  mail  it  promptly.  Your  response  is 
essential  to  the  success  of  this  research  effort.  If  you 
have  any  questions  or  you  would  like  to  obtain  a  copy  of  the 
results,  please  contact  me  at  the  following  address: 

Captain  Jon  Congdon 
AFIT/LSG  (GSM88S) 

Wright-Patterson  AFB,  OH  45433 
Office  Phone:  (513)  255-6569 


THANK  YOU  FOR  YOUR  TIME  AND  COOPERATION. 


87 


Part  I 


The  following  is  a  list  of  formal  project  selection 
techniques.  If  you  are  familiar  with  the  technique,  please 
place  an  "XM  in  the  appropriate  column.  If  you  also  use  the 
technique  regularly  in  your  project  selection  process,  please 
mark  the  appropriate  column.  If  you  are  not  familiar  with 
the  technique,  please  do  not  mark  either  column. 

Technique  Familiar  Use 

With  Regularly 


1 .  Q-Sort 

2.  Checklists 

3.  Scoring  Models 

4 .  Delphi 

5.  Cost/Benefit  Ratios 

6 .  Payback  Period 

7.  Net  Present  Value/ 
Internal  Rate  of  Return 

8.  Portfolio  Models 

9.  Risk  Analysis 

(Monte  Carlo  Simulation) 

10.  Decision  Analysis/ 
Decision  Trees 

11.  Linear  Programming 

12 .  Integer  Programming 

13.  Nonlinear  Programming 

14 .  Dynamic  Programming 

15.  Goal  Programming 

16.  Others?  Please  list 
below. 
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For  the  following  questions,  please  place  an  "X"  inside 
the  box  that  most  closely  matches  your  opinion.  The  use  of 
the  term  "formal  project  selection  techniques"  refers  to 
techniques  such  as  those  listed  in  questions  1  through  16. 

17.  How  would  you  assess  the  amount  of  influence  that  you 
have  on  your  organization's  R&D  project  selection  and 
resource  allocation  process? 


Very 

strong 

influence 


Strong 

influence 


Some 

influence 


Little 

influence 


No 

influence 


18..  How  would  you  assess  the  adequacy  of  the  manner  in  which 
your  division  reviews  and  evaluates  potential  R&D  projects? 


Could  be 
improved  a 
great  deal 


Could  be 
improved 
somewhat 


Uncertain 


Probably 
could  not 
be  improved 


Could 
not  be 
improved 


19.  How  would  you  assess  the  potential  worth,  vou 
personally,  of  a  technique  which  would  be  designed  to  assist 
you  in  measuring  the  comparative  value  of  potential  R&D 
proj ects? 


Extremely 

bene¬ 

ficial 


Somewhat 

bene¬ 

ficial 


No 

benefit  or 
uncertain 


Somewhat 

detri¬ 

mental 


Extremely 

detri¬ 

mental 


1 

2 

3 

4 

5 

6 

7 
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20.  How  would  you  assess  the  potential  worth,  £2  those 
people  with  whom  vou  work,  of  a  technique  which  would  be 
designed  to  assist  them  in  measuring  the  comparative  value 
of  potential  R&D  projects? 


Extremely  Somewhat  No  Somewhat  Extremely 

bene-  bene-  benefit  or  detri-  detri- 

ficial  ficial  uncertain  mental  mental 


21.  How  important,  £2  vou  personally,  would  you  consider  a 
decision  to  use  a  formal  project  selection  technique  on  all 
your  R&D  projects? 

Very  Somewhat  Neither  Somewhat  Very 

important  important  important  unimportant  unimportant 
to  you  to  you  nor  unim-  to  you  to  you 

portant 


22.  If  such  a  decision  were  to  be  made  what  would  you 
recommend? 


Definitely  Probably  Uncertain  Probably  Definitely 

should  should  should  should 

use  use  not  use  not  use 


23.  How  important,  £2  you  personally,  would  you  consider  a 
decision  to  use  a  formal  project  selection  technique  on  at 
least  50  percent  of  your  R&D  projects? 


Very  Somewhat  Neither  Somewhat  Very 

important  important  important  unimportant  unimportant 
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24.  If  such 

a  decision 

were  to  be  made,  what  would  you 

recommend? 

Definitely 

Probably 

Uncertain  Probably 

Definitely 

should 

should 

should 

should 

use 

use 

not  use 

not  use 

25.  Do  you  feel  it  is  likely  that  a  formal  project  selection 
technique  could  be  developed  which  would  be  useful  to  you 
during  your  project  selection  and  resource  allocation 
process? 


Very 

Likely 


Likely 


Uncertain  Unlikely 


Very 

Unlikely 


26.  To  what  extent  do  you  feel  "pressure"  from  outsid< 
AFWAL  to  quantitatively  justify  your  R&D  expenditures? 


Extreme 

pressure 


Significant 

pressure 


Some 

pressure 


Little 

pressure 


No 

pressure 


27.  To  what  extent  do  you  feel  "pressure"  from  within  AFWAL 
to  quantitatively  justify  your  R&D  expenditures? 


Extreme 

pressure 


Significant 

pressure 


Some 

pressure 


Little 

pressure 


No 

pressure 


28.  In  the  space  below,  please  describe  any  positive  or 
negative  experiences  you've  had  when  using,  or  attempting  to 
use,  a  formal  project  selection  technique. 


Part  II 


Each  item  in  this  portion  of  the  survey  consists  of  a 
statement  with  which  you  are  asked  to  agree  or  disagree. 
Using  the  scale  shown  below,  please  indicate  the  extent  to 
which  you  agree  or  disagree  with  each  statement.  Place  a 
number  which  corresponds  to  your  opinion  in  the  space 
provided.  Please  indicate  your  opinion  even  though  you  may 
not  feel  strongly  about  the  statement,  or  do  not  feel  well 
informed.  Only  your  initial  perceptions  are  sought.  Please 
try  to  avoid  marking  a  "3"  (Cannot  agree  or  disagree)  if 
possible. 


Strongly 

agree 

— 

Tend  to 
agree 

Can't  agree 
or  disagree 

Tend  to 
disagree 

Strongly 

disagree 

1 

2 

3 

4 

5 

Using  the  scale  shown  above,  please  place  a  number  in  the 

space  provided  which  corresponds  to  your  opinion  about  each 

statement . 

_  29.  I  am,  in  general,  very  satisfied  with  the 

manner  in  which  projects  and  tasks  are 
reviewed  and  evaluated  in  our  division. 

_  30.  The  need  for  a  formal  project  selection 

technique  is  not  great  relative  to  other 
changes  that  could  be  made  in  our  laboratory's 
project  review  and  budget  allocation  process. 

_  31.  My  assessment  of  particular  projects  or  tasks 

is  not  likely  to  change  as  a  result  of  using  a 
formal  project  selection  technique. 

_  32.  A  formal  project  selection  technique  would 

help  to  make  my  budget  recommendations  more 
compatible  with  those  of  my  superiors . 

_  33.  A  formal  project  selection  technique  would 

help  to  make  my  budget  recommendations  more 
compatible  with  those  of  my  research  staff. 

_  34.  A  formal  project  selection  technique  is  likely 

to  increase  the  confidence  I  have  in  my  budget 
recommendations . 


Strongly 

agree 

Tend  to 
agree 

Can't  agree 
or  disagree 

Tend  to 
disagree 

Strongly 

disagree 

1 

2 

3 

4 

5 

Using  the  scale  shown  above,  please  place  a  number  in  the 

space  provided  which  corresponds  to  your  opinion  about  each 

statement . 

_  35.  A  formal  project  selection  technique  will 

affect  the  type  of  information  exchanged  when 
I  review  and  evaluate  the  projects  and  tasks 
in  my  division/branch. 

_  36.  A  formal  project  selection  technique  will 

affect  the  amount  of  information  exchanged 
when  I  review  and  evaluate  the  projects  and 
tasks  in  my  division/branch. 

_  37.  The  manner  in  which  we  review  and  evaluate 

projects  and  tasks  in  our  division  does  not 
lend  itself  to  the  use  of  a  formal  project 
selection  technique. 

_  38.  Formal  project  selection  techniques  are,  in 

general,  difficult  to  use  and  understand. 

_  39.  The  information  generated  by  a  formal  project 

selection  technique  cannot  be  easily  communi¬ 
cated  to  others. 

_  40.  My  use  of  a  formal  project  selection  technique 

is  likely  to  enhance  the  worth  of  jay  budget 
recommendations . 

_  41.  My  use  of  a  formal  project  selection  technique 

is  likely  to  be  considered,  by  my  immediate 
superiors .  as  having  enhanced  the  worth  of  my 
budget  recommendations. 

_  42.  My  use  of  a  formal  project  selection  technique 

is  likely  to  be  considered,  by  my  research 
staff,  as  having  enhanced  the  worth  of  my 
budget  recommendations. 

_  43.  More  relevant  information  is  likely  to  be 

exchanged  in  my  division/branch  if  I  use  a 
formal  project  selection  technique. 
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Strongly 

agree 

Tend  to 
agree 

Can't  agree 
or  disagree 

Tend  to 
disagree 

Strongly 

disagree 

1 

2 

3 

4 

5 

Using  the  scale  shown  above,  please  place  a  number  in  the 

space  provided  which  corresponds  to  your  opinion  about  each 

statement. 

' _  44.  A  formal  project  selection  technique  is 

likely  to  help  identify  critical  or  sensitive 
aspects  of  my  projects  or  tasks. 

_  45.  My  use  of  a  formal  project  selection  technique 

is  likely  to  increase  the  overall  effective¬ 
ness  of  reviewing  and  evaluating  our 
division's  projects  and  tasks. 

_  46.  The  output  data  that  would  be  generated  by  a 

formal  project  selection  technique  could  not 
adequately  meet  my  need  for  such  data. 

_  47.  I  have  some  reservation  about  the  way  in  which 

formal  project  selection  techniques  combine  or 
manipulate  the  input  data. 

_ _  48.  I  would  find  it  extremely  difficult  to  obtain 

the  input  data  necessary  to  use  a  formal 
project  selection  technique. 

_  49.  The  need  for  a  formal  project  selection 

technique  is  very  much  increased  when  funds 
are  scarce. 

_  50.  I  would  recommend  that  a  formal  project 

selection  technique  be  used  on  all  of  the 
projects  or  tasks  in  my  division. 

_  51.  I  would  recommend  that  a  formal  project 

selection  technique  be  used  on  least  50 
percent  of  the  projects  or  tasks  in  my 
division. 

_  52.  The  process  by  which  most  formal  project 

selection  techniques  generate  their  output 
data  could  easily  be  explained  to  my  col¬ 
leagues. 
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Using  the  scale  shown  above,  please  place  a  number  in  the 

space  provided  which  corresponds  to  your  opinion  about  each 

statement . 

_  53.  I  personally  feel  that  R&D  project  selection 

decisions  which  are  based  on  experience  and 
expertise  will  yield  better  results  than 
decisions  made  with  the  use  of  a  formal 
project  selection  technique. 

_  54.  The  output  data  generated  by  a  formal  project 

selection  technique  could  be  easily  inter¬ 
preted  by  laboratory  decision  makers. 

_  55.  It  is  likely  that  a  formal  project  selection 

technique  will  provide  common  outputs  that 
could  be  easily  understood  by  all  laboratory 
decision  makers. 

_  56.  The  mathematics  involved  in  most  formal 

project  selection  techniques  are  more 
complicated  than  is  necessary  for  our  project 
selection  process. 

_ _  57.  The  input  data  for  most  project  selection 

techniques  is  difficult  to  generate  due  to  the 
uncertainties  associated  with  technological 
developments . 

_  58.  When  I  make  R&D  project  selection  decisions, 

I  prefer  to  use  a  structured,  quantifiable 
process . 


59. 

What  is 

your 

sex? 

(Circle  one) 

A) 

Male 

B) 

Female 

60. 

What  is 

your 

age? 

A) 

20-25 

B) 

26-30 

C) 

31-35 

(Circle 

one) 

D) 

36-40 

E) 

41-45 

F) 

46-50 

G) 

51-55 

H) 

56  + 

61.  Please  print  your  military  rank  or  civilian  grade. 


62.  Which  AFWAL  organization  are  you  associated  with? 
(Circle  one) 

A)  Avionics  Laboratory 

B)  Flight  Dynamics  Laboratory 

C)  Material  Laboratory 

D)  Aero  Propulsion  Laboratory 

E)  AFWAL  Chief  of  Staff  Section 


63.  What  is  the  highest  educational  level  you  have  obtained? 
(Circle  one) 

A)  High  School  Diploma 

B)  Associate's  Degree 

C)  Bachelor's  Degree 

D)  Master's  Degree 

E)  Ph.D. 


64.  What  were  your  major  fields  of  study  in  the  degrees  you've 
obtained? 


A)  Associate's  Degree 

B)  Bachelor's  Degree 

C)  Master's  Degree 

D)  Ph.D. 


65.  Have  you  ever  taken  a  course,  or  attended  a  seminar  in 

either  management  science  or  operations  research  techniques? 
(Circle  one) 

A)  Yes  B)  No 
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66.  Would  you  be  willing  to  attend  a  course  or  seminar  on  formal 
R&D  project  selection  techniques?  (Circle  one) 

A)  Yes  B)  No 


67.  Have  you  ever  taken  a  course  or  seminar  in  R&D  or  laboratory 
management?  (Circle  one) 

A)  Yes  B)  No 


68.  What  is  the  approximate  size  of  the  budget  you  are 
responsible  for? 

$  _ . 


69.  What  is  your  estimate  of  the  percentage  of  your  division's 
budget  dedicated  to  in-house  research  versus  contracted 
research? 

In-house  %  _  Contracted  %  _  (Total  100%) 


70.  What  is  your  estimate  of  the  percentage  of  your  budget  that 
is  dedicated  to  discretionary  research  efforts  which  are 
chosen  by  your  division  for  their  potential  value  to  future 
Air  Force  weapons  systems,  as  opposed  to  research  efforts 
which  are  specifically  requested  by  an  Air  Force  system 
program  office  (SPO)  or  another  outside  organization  to 
support  a  particular  weapons  system? 

Discretionary  research 

chosen  by  you  or  your 

division  %  _ 

Research  specifically 
requested  by  a  SPO 

or  outside  organization  %  _  (Total  100%) 


THE  BMP 


THANK  YOU  FOR  TAKING  THE  TIME  TO  COMPLETE  THIS  SURVEY 
YOUR  COOPERATION  IS  GREATLY  APPRECIATED 
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Appendix  B:  Responses  to  Survey  Question  28 

1.  I  have  used  both  Delphi  and  Decision  Trees.  The  largest 
problem  is  trying  to  achieve  a  process  that  is  applicable 
across  the  board.  Since  I'm  in  the  Avionics  Lab,  and  since 
our  diversity  is  so  great,  it  is  difficult  to  prioritize  one 
generic  technology  against  another.  As  things  become  more 
specific,  it  becomes  easier  to  say  which  degree  of  specificity 
should  be  ranked  higher. 

2.  The  difficulty  lies  in  having  a  model  flexible  enough  to 
use  on  a  variety  of  programs.  R&D  programs  tend  to  vary 
greatly  in  risk  (cost  and  technology) ,  payback,  direct 
application  to  current  systems,  etc.. 

3.  It  is  difficult  to  substitute  some  system  of  numbers  for 
judgment.  In  fact,  somewhere  in  a  number s-assigning 
process,  one  must  exercise  judgement.  Managers  use  favorable 
numbers  to  justify  what  they've  already  decided  to  do,  or 
ignore  unfavorable  numbers  and  go  ahead  with  what  they've 
already  decided  to  do. 

4.  We  have  a  formal  project  selection  technique  which  we 
have  used  and  found  to  be  quite  effective  for  many  years. 

5.  Vision.  One  must  have  a  vision  for  R&D  work,  and  must 
be  willing  and  able  to  fund  and  support  that  vision.  Formal 
project  selection  techniques  can  be  useful,  but  never  as  an 
absolute  or  single  decision  process  -  use  it  as  a  tool.  It 
is  very  important  to  get  folks  to  use  their  minds,  their 
vision,  and  their  judgment  about  the  potential  of  projects, 
the  plan  for  accomplishing  their  projects,  and  the  resources 
applied  (including  the  leader  and  the  timing) .  Obviously 
such  factors  should  go  into  a  project  selection  technique 
and  thus  we  should  use  it.  However,  one  must  still  be 
willing  and  able  to  use  that  as  an  input,  and  can't  depend 
on  a  subordinate  group  to  do  all  the  thinking  or  to  fill  in 
the  blanks  by  rote.  You  can't  have  a  useful  technique  that 
does  not  put  a  premium  on  experience  and  expertise. 

6.  I  designed  an  off-site  where  we  were  going  to  quantify 
"cutting  edge  technologies"  using  a  "cross-impact  matrix" 
(after  Bennis  &  Manus) ,  and  a  voting  scheme.  Results  were 
accumulated,  and  would  have  been  used  to  decide  what 
programs  to  fund  and  not  to  fund.  When  it  was  discovered 
what  the  results  of  the  voting  would  do  -  that  is,  potentially 
kill  "gold  watches"  -  the  process  was  abandoned. 

7.  They  are  typically  oversold  in  terms  of  their  capability. 
The  informal  process  usually  provides  answers  that  are 
accurate  to  within  90-95%  of  more  formal  processes. 
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8.  The  Delphi  process  is  useful  to  focus  on  similarities 
and  differences  of  opinion. 

9.  Have  never  used  a  formal  project  selection  technique. 

10.  It  is  very  difficult  to  communicate  to  higher  management. 
Very  hard  to  break  away  from  the  influence  of  senior  civilian 
and  military  managers  who  feel  their  experience  is  sufficient 
to  make  judgments. 

11.  Formal  project  selection  processes  are  used  now  with 
considerable  success.  The  process  is  not  computer  automated; 
it  is  not  a  by-the-numbers  algorithm,  and  it  should  not  be, 
though  it  could  use  some  of  these  elements. 

12.  We  have  an  excellent  process,  developed  over  a  period  of 
many  years,  for  the  planning,  prioritization,  and  selection 
of  our  programs.  To  impose  a  new,  mandatory  process  would  be 
very  counterproductive.  We  do  not  need  more  rigidity,  more 
accounting,  more  process.  We  need  more  work,  more  time  to  do 
it,  and  for  people  at  all  levels  to  be  personally  accountable 
for  their  results .  not  their  plans  or  their  promises. 

13.  Quantifying  payoffs,  working  to  AF  needs,  etc.,  is  a  way 
of  life  to  the  technology  community.  Formal  processes 
(techniques)  in  my  experience  have  not  survived  because  they 
attempt  to  normalize  the  vision  of  the  technologist.  The 
strength  of  the  technology  base  is  the  in-depth  technical 
knowledge,  creativity,  and  innovations  of  the  scientists  and 
engineers,  coupled  with  their  understanding  and  perception 
of  the  current  and  future  military  work  in  their  technical 
area.  Technology  push  vs.  requirements  pull,  in-house  vs. 
contract,  facility  upgrades,  people,  etc.,  are  all  parts  of 
the  investment  strategy  that  have  to  be  addressed.  The  use 
of  selection  techniques  for  exploratory  development  (6.2) 
and  basic  research  (6.1),  is  of  limited  value  while  for 
advanced  development  (6.3)  and  beyond,  the  value  should 
increase.  My  experience  suggests  that  the  formal  selection 
techniques  normally  suffer  from  processing  very  subjective 
inputs  to  output  quantified  decision  and  priority  data. 

Using  and  fine  tuning  a  technique  over  a  long  time  period 
(years)  may  result  in  a  useful  tool.  The  success  of  future 
project  selection  techniques  applied  to  technology  could 
benefit  from  an  AI  (Artificial  Intelligence)  based  approach 
using  the  decision  process  that  technologists  go  through  in 
developing  and  defining  their  own  technology  programs. 

14.  In  the  past,  AFSC  attempted  to  standardize  and  formalize 
a  process  of  rating  bidder  proposals  to  do  R&D  programs.  It 
had  the  acronym  TORQUE.  It  was  a  disaster,  and  was  left  to 
die  without  a  formal  cancellation.  I  think  any  formal 
technique  of  rating  anything  is  bad! 
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15.  The  time  it  usually  takes  to  do  mathematical  computations 
is  usually  too  long. 

16.  A  list  of  criteria  (weighted)  has  been  used  as  a 
template  on  various  occasions.  The  criteria  have  varied 
based  on  the  exercise.  Near  term/ far  term  comparisons  were 
use  in  the  selection  process.  Absolute,  strict  use  was 
"disturbing1*  because  some  programs  that  we  knew  "should"  be 
done  didn't  fit  the  "template." 

17.  The  Materials  Laboratory  has  a  detailed  budget  process 
based  on  "focal  points."  This  process  does  not  fit  your 
formal  technique,  but  is  very  productive.  AFWAL  uses  a 
formal  technique  for  their  POM  submission.  It  is  of  some 
value,  however,  you  still  need  a  great  deal  of  judgment  in 
such  a  process. 

18.  Any  project  selection  criteria  must  be  based  on  a  strong 
set  of  requirements  (i.e.  technical,  financial,  needs  of  the 
AF)  . 


19.  Project  selection  is  dependent  on  many  criteria,  few  of 
which  can  be  quantified.  Previous  experience  has  shown  that 
various  selection  techniques  lead  to  demonstrably  wrong 
decisions.  Also,  the  worth  of  certain  technologies  varies 
through  the  project  lifetime.  It  is  difficult  to  apply 
selection  methodologies  which  have  been  devised  for  other 
types  of  projects. 

20.  Past  experiences  have  shown  that  for  6.1  and  6.2 
endeavors,  formalized  project  selection  techniques  provided 
little,  if  any,  added  value  to  the  final  decision  process. 
Rather,  sound  technological  trending,  based  on  experienced 
scientific  and  engineering  inputs,  proves  the  primary  basis 
for  project  selection. 

21.  Past  attempts  at  R&D  project  selection  (such  as  TORQUE 
or  RDE)  suffer  from  the  same  problem.  They  do  not  evaluate 
a  project  in  the  context  of  its  application.  Rather,  the 
assumption  seems  to  be  that  an  individual  program  can  be 
evaluated  as  an  isolated  entity.  This  cannot  work  because 
the  value  of  a  specific  technological  contribution  lies  in 
its  marginal  contribution  to  the  next  level  of  aggregation 
or  integration  compared  to  other  options.  This  requires 
dynamic  programming  and  hierarchical  modelling,  and  is 
rather  labor  intensive.  At  the  HQ  AFWAL  level,  we  have  used 
Delphi  to  rank  order  biennial  POM  cost/benefit  packages  for 
over  ten  years.  It  works  extremely  well. 
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Appendix  C: 


1 .  Name : 

2.  Rank/Grade: 

3.  Organization: 

4.  How  does  your  organization  evaluate  and  select  research 
projects  during  your  resource  allocation  process?  Does  your 
organization  utilize  a  standardized  process?  If  so,  does 
this  process  rely  on  the  use  of  formal  project  selection 
techniques? 

5.  Who  participates  in  this  process?  What  roles  do  they 
have? 

6.  Does  this  process  differ  depending  on  the  type  of 
research  being  conducted  (e.g.  6.1,  6.2,  6.3)?  If  so,  how? 

7.  Do  you  feel  a  formal  project  selection  process  would 
either  enhance  or  diminish  the  effectiveness  of  your  project 
selection  process?  Please  explain. 

8.  How  do  budget  cutbacks  affect  your  project  selection 
process? 

9.  Do  you  feel  any  pressure  to  quantitatively  justify  your 
R&D  project  selections?  Please  explain. 

10.  What  characteristics  do  you  feel  a  useful  project 
selection  process  should  have? 

11.  What  criteria  do  you  feel  are  important  to  consider 
when  reviewing  R&D  projects? 

12.  What  comments  would  you  like  to  add  concerning  the  R&D 
project  selection  process  or  any  other  aspect  of  this 
research? 
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implementation  of  R&D  project  selection  techniques  at  the 
Air  Force  Wright  Aeronautical  Laboratories  (AFWAL) .  It 
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selection  decisions;  2)  the  attitudes  and  perceptions  of 
managers  toward  the  utility  of  R&D  project  selection  techni¬ 
ques;  and  3)  the  specific  factors  affecting  the  adoption  of 
these  techniques  by  R&D  managers. 

Findings  showed  that  R&D  project  selection  techniques 
play  a  minor  role  in  the  project  selection  process,  and  that 
managers  are  generally  unfamiliar  with  these  techniques. 
Instead,  project  selection  decisions  are  often  made  by  top- 
level  laboratory  managers  through  a  review  process.  Findings 
also  showed  that  the  project  selection  process  at  AFWAL  is 
primarily  driven  by  "requirements  pulit"  as  opposed  to 
"technology  push.*1* 

Factor  analysis  was  used  to  identify  an  initial  set  of 
variables  which  affect  management's  willingness  to  adopt  R&D 
project  selection  techniques.  These  variables  were  factored 
into  two  groups  relating  to  1)  the  impact  of  formal  project 
selection  techniques  on  organizational  decision-making;  and 
2)  the  characteristics  of  the  project  selection  techniques 
themselves.  The  results  showed  that  the  respondents 
possessed  negative  perceptions  toward  the  utility  of  these 
techniques. 

Previous  research  efforts  have  shown  that  negative-to- 
positive  attitude  shifts  have  occurred  when  respondents  were 
provided  with  education  and  training  in  the  use  of  these 
techniques.  Consequently,  it  was  recommended  that  managers 
be  provided  with  training  to  increase  their  understanding  of 
these  techniques.  A  suggestion  was  also  made  to  examine  the 
applicability  of  computer-based  decision  support  systems  to 
the  R&D  project  evaluation  and  selection  process. 


